How to remove items from an ansible list

In my Ansible playbook, I need to change permissions on a lot of files, but a few subdirectories need read-write permissions, while most need read-only. Following another suggestion from superuser, I have this solution:

- name: A few directories need group-write permissions
  file:
    path: "{{item}}"
    mode: "u+rwX,g+rwX,o+rX,o-w"
    recurse: True
  with_items:
    - /opt/myapp/path1/excludeddir1
    - /opt/myapp/path1/excludeddir2
    - /opt/myapp/path2/excludeddir1
    - /opt/myapp/path2/excludeddir2

- name: For performance, set a lot of directories directly
  file:
    path: "{{item}}"
    mode: "u+rwX,go+rX,go-w"
    recurse: True
  with_items:
    - /opt/myapp/path1/readonlydir1
    - /opt/myapp/path1/readonlydir2

#############################################
# This step generates a very large list
- name: Find all files in my directory
  find:
    paths:
      - "/opt/myapp/path1"
      - "/opt/myapp/path2"
    recurse: True
    file_type: any
    follow: False
  register: filestochange

#############################################
# This step is painfully slow
- name: Clear group-write permissions where needed
  file:
    path: "{{ item.path }}"
    mode: "u+rwX,go+rX,go-w"
    follow: False
  when:
    - not item.islnk
    - "'/opt/myapp/path1/excludeddir1' not in item.path"
    - "'/opt/myapp/path1/excludeddir2' not in item.path"
    - "'/opt/myapp/path2/excludeddir1' not in item.path"
    - "'/opt/myapp/path2/excludeddir2' not in item.path"
    - "'/opt/myapp/path1/readonlydir1' not in item.path
    - "'/opt/myapp/path1/readonlydir2' not in item.path
  loop: "{{ filestochange.files }}"
  loop_control:
    label: "{{ item.path }}"

The subdirectories involved have a total of approximately 100k files.

The above code works, but is, not surprisingly, painfully slow. The original mostly naive implementation ran for two days straight.

My first optimization was to set the permissions for a few subdirectories that don't need any exceptions, and later skip them in the loop. This helped to some extent; the time is now down to an hour or two.

As a next optimization, I would like to remove those same entries from the list that the find module generates before even feeding it into the loop, but I have not found a way to do that.

The file module does have an exclude attribute, but that seems to only match file names, not directory names.

So I am looking for a way to remove items from a list that match certain wildcard patterns.

Of course I'm also open to any other suggestions how to further optimize this.

Note: this naive implementation does NOT work as it would reset, and then set, the permissions.

- name: Set everything to G read-only
  file:
    path: "{{item}}"
    mode: "u+rwX,go+rX,go-w"
    recurse: True
  with_items:
    - /opt/myapp/path1
    - /opt/myapp/path2

- name: A few directories need group-write permissions
  file:
    path: "{{item}}"
    mode: "u+rwX,g+wrX,o+rX,o-w"
    recurse: True
  with_items:
    - /opt/myapp/path1/excludeddir1
    - /opt/myapp/path1/excludeddir2
    - /opt/myapp/path2/excludeddir1
    - /opt/myapp/path2/excludeddir2