Why does xargs -I imply -L 1

gnuhistoryxargs

I've seen Why does 'find -exec cmd {} +' need to end in '{} +'? which explains why {} + has to be at the end of the -exec cmd(*), and now I'd like to know why GNU xargs reverts to one command per input argument when you use -I (or the deprecated -i)…or as the man page puts it

-I replace-str […] Implies -x and -L 1.

For example:

$ seq 1 10 | xargs echo
1 2 3 4 5 6 7 8 9 10
$ seq 1 10 | xargs -I {} echo {}
1
2
3
4
5
6
7
8
9
10

The BUGS section of the xargs man page only says:

The -L option is incompatible with the -I option, but perhaps should
not be.

and

When you use the -I option, each line read from the input is
buffered
internally. This means that there is an upper limit on the length of
input line that xargs will accept when used with the -I option. To
work around this limitation, you can use the -s option to increase the
amount of buffer space that xargs uses, and you can also use an extra
invocation of xargs to ensure that very long lines do not occur.

Neither of which explains why.

One of the main reasons to use xargs is to reduce the number of commands executed (running one cp or mv with 1000 source filenames is better than running 1000 cps or mvs with 1 source filename each) and this limitation prevents that in many cases….most cases since most unix commands want the source (e.g. a list of files) before the destination (e.g. a directory).

So, why does xargs have this limitation?


Update 2015-11-05

I recently created a freebsd 10.0 VM and found that freebsd's version of xargs has a -J option to deal with this issue.

-J replstr

If this option is specified, xargs will use the data read from standard input to replace the first occurrence of replstr instead of appending that data after all other arguments. This option will not affect how many arguments will be read from input (-n), or the size of the command(s) xargs will generate (-s). The option just moves where those arguments will be placed in the command(s) that are executed. The replstr must show up as a distinct argument to xargs. It will not be recognized if, for instance, it is in the middle of a quoted string. Furthermore, only the first occurrence of the replstr will be replaced. For example, the following command will copy the list of files and directories which start with an uppercase letter in the current directory to destdir:

/bin/ls -1d [A-Z]* | xargs -J % cp -Rp % destdir

(*) Essentially, "because the POSIX spec says so". IMO they should have tried harder to find a solution that allowed {} to appear anywhere in the -exec ... + , or at least required standard tools like cp and mv to have an option like GNU's -t option to reverse source and target.

Best Answer

(As this does not answer the question, this is should have been a comment, but is too long - so treat it as a comment).

As an alternative to FreeBSD's xargs you can use GNU Parallel which does not have this limitation. It even supports repeating the context:

seq 10 | parallel -Xj1 echo con{}text 
seq 10 | parallel -mj1 echo con{}text 

GNU Parallel is a general parallelizer and makes is easy to run jobs in parallel on the same machine or on multiple machines you have ssh access to. It can often replace a for loop.

If you have 32 different jobs you want to run on 4 CPUs, a straight forward way to parallelize is to run 8 jobs on each CPU:

Simple scheduling

GNU Parallel instead spawns a new process when one finishes - keeping the CPUs active and thus saving time:

GNU Parallel scheduling

Installation

If GNU Parallel is not packaged for your distribution, you can do a personal installation, which does not require root access. It can be done in 10 seconds by doing this:

(wget -O - pi.dk/3 || curl pi.dk/3/ || fetch -o - http://pi.dk/3) | bash

For other installation options see http://git.savannah.gnu.org/cgit/parallel.git/tree/README

Learn more

See more examples: http://www.gnu.org/software/parallel/man.html

Watch the intro videos: https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1

Walk through the tutorial: http://www.gnu.org/software/parallel/parallel_tutorial.html

Sign up for the email list to get support: https://lists.gnu.org/mailman/listinfo/parallel

Related Question