How to Limit Unix Find Results for Directories with Many Files

You could try without find, if all you want is all the files. While in data/all/, run this:

for file in ./*; do awk '{print$2}' "$file" > "../pos/$(basename $file)"; done

If you want to cover the files in the whole hierarchy under /data/all, you can enable the globstar option if you are using bash (I believe this would "just work" on zsh) and then use ** to match all files:

shopt -s globstar
for file in ./**; do awk '{print$2}' "$file" > "../pos/$(basename $file)"; done

My attempt:

chmodx() (
   type="$1"
   shift
   cleared=
   marked=
   recursive=
   for a do
      if [ -z "$cleared" ]; then
         set -- -type "$type" -exec chmod
         cleared=y
      fi
      if [ -z "$marked" ]; then
         case "$a" in
         -- )
            set -- "$@" -- {} +
            if [ -z "$recursive" ]; then
               set -- -prune "$@"
            fi
            marked=y
            ;;
         -R|--recursive )
            recursive=y
            ;;
         * )
            set -- "$@" "$a"
            ;;
         esac
      else
         set -- "$a" "$@"
      fi
   done
   if [ -z "$marked" ]; then
      printf '`--'"'"' required\n'
      return 1
   fi
   exec find "$@"
)

alias chmodf='chmodx f'
alias chmodd='chmodx d'

The function and aliases turn

chmodf  args for chmod  --  starting points

into

find  points starting  -type f -exec chmod  args for chmod  {} +

Similarly chmodd … turns into find … -type d ….

It's mainly about rearranging arguments. There is almost no parsing. Everything before -- is treated as args for chmod. Everything after -- is treated as starting points for find. We don't want to actually run chmod recursively, but we may or may not run find recursively. Non-recursive operation is performed thanks to -prune in the right place. If there is -R or --recursive in args for chmod then it will disappear but -prune will not appear.

Notes:

• I think the code is portable. I deliberately avoided using arrays.

• chmodf and chmodd could be functions. Even then they should call chmodx to keep the code DRY. Implementing them as two independent functions is inelegant.

• -- is mandatory because I decided to KISS. It's certainly possible to tell apart pathnames from other arguments without separating them with --. Actually chmod does this. It knows its own syntax, all its possible options and such; so everything else must be files to operate on. Embedding this knowledge and functionality in a shell script is WET. If you really want things to work without -- then you should rather solve your problem by improving chmod, modifying its source.

• There are ways to abuse:

  • My code does not validate starting points, so you can inject e.g. -maxdepth 3 this way (as 3 -maxdepth because starting points become points starting).
  • Similarly there is no validation of args for chmod. If args for chmod include ; or {} + then the -exec primary will be terminated early in the line. This allows you to inject more primaries; or to break the whole command by generating invalid syntax.

  While technically possible, such tricks are ugly. If I needed to complicate things, I would definitely prefer to write a find command anew.