Makefile – Fix ‘Argument List Too Long’ Error

argumentsmakeposix

In a makefile, I have

@echo "$(IGNORE_DIRS) $(CLEAN_FILES) $(CLEAN_DIRS) $(REALCLEAN_FILES)" | tr ' ' '\n' >> $@

The problem is that $(CLEAN_FILES) is quite large, so when I run make, I get

make: execvp: /bin/sh: Argument list too long

I'm on Xubuntu 18.10.

Edit: I should provide a little more context. What I am working on is a make rule (I'm using GNU make) to automatically generate the .hgignore file. Here is the make rule in its entirety:

.hgignore : .hgignore_extra
    @echo "Making $@"
    @rm -f $@
    @echo "# Automatically generated by Make. Edit .hgignore_extra instead." > $@
    @tail -n +2 $< >> $@
    @echo "" >> $@
    @echo "# The following files come from the Makefile." >> $@
    @echo "syntax: glob" >> $@
    @echo "$(IGNORE_DIRS) $(CLEAN_FILES) $(CLEAN_DIRS) $(REALCLEAN_FILES)" | tr ' ' '\n' >> $@
    @chmod a-w $@
.PHONY : .hgignore

Edit 2: At @mosvy 's suggestion, I have also tried

.hgignore : .hgignore_extra
    @echo "Making $@"
    @rm -f $@
    @echo "# Automatically generated by Make. Edit .hgignore_extra instead." > $@
    @tail -n +2 $< >> $@
    @echo "" >> $@
    @echo "# The following files come from the Makefile." >> $@
    @echo "syntax: glob" >> $@
    $(file >$@) $(foreach V,$(IGNORE_DIRS) $(CLEAN_FILES) $(CLEAN_DIRS) $(REALCLEAN_FILES),$(file >>$@,$V))
    @true
    @chmod a-w $@
.PHONY : .hgignore

Running make .hgignore with this, I no longer get the "Argument list too long" error, but the generated .hgignore file only contains output up to the syntax: glob line, and then nothing after that.

Best Answer

As @schily has already explained, this is not a shell problem, and cannot be worked around with xargs, quoting, splitting into more echo's with ;, etc. All the text from a make action is passed as argument/s to a single execve(2), and it can't be longer than the maximum size allowed by the operating system.

If you're using GNU make (the default on linux), you can use its file and foreach functions:

TEST = $(shell yes foobar | sed 200000q)

/tmp/junk:
        $(file >$@) $(foreach V,$(TEST),$(file >>$@,$V))
        @true

.PHONY: /tmp/junk

This will print all words from $(TEST) separated by newlines into the file named in $@. It's based on a similar example from make's manual.

Your Makefile could probably be reworked into something more manageable, that doesn't require fancy GNU features, but it's hard to tell how from the snippets you posted.

Update:

For the exact snippet from the question, something like this could do:

.hgignore : .hgignore_extra
    $(info Making $@)
    $(file >$@.new)
    $(file >>$@.new,# Automatically generated by Make. Edit .hgignore_extra instead.)
    $(shell tail -n 2 $< >>$@.new)
    $(file >>$@.new,)
    $(file >>$@.new,# The following files come from the Makefile.)
    $(file >>$@.new,syntax: glob)
    $(foreach L, $(IGNORE_DIRS) $(CLEAN_FILES) $(CLEAN_DIRS) $(REALCLEAN_FILES), $(file >>$@.new,$L))
    @mv -f $@.new $@
    @chmod a-w $@
.PHONY : .hgignore

I've changed it a little, so it first writes into .hgignore.new, and if everything goes well, only then move .hgignore.new to .hgignore. You'll have to change back the indenting spaces to tabs, because this dumb interface is mangling whitespaces.

Related Solutions

ls – Argument List Too Long for ls Command

Your error message argument list too long comes from the ***** of ls *.txt.

This limit is a safety for both binary programs and your Kernel. See ARG_MAX, maximum length of arguments for a new process for more information about it, and how it's used and computed.

There is no such limit on pipe size. So you can simply issue this command:

find -type f -name '*.txt'  | wc -l

NB: On modern Linux, weird characters in filenames (like newlines) will be escaped with tools like ls or find, but still displayed from *****. If you are on an old Unix, you'll need this command

find -type f -name '*.txt' -exec echo \;  | wc -l

NB2: I was wondering how one can create a file with a newline in its name. It's not that hard, once you know the trick:

touch "hello
world"

Bash Shell – Solving ‘mv: Argument List Too Long’

xargs is the tool for the job. That, or find with -exec … {} +. These tools run a command several times, with as many arguments as can be passed in one go.

Both methods are easier to carry out when the variable argument list is at the end, which isn't the case here: the final argument to mv is the destination. With GNU utilities (i.e. on non-embedded Linux or Cygwin), the -t option to mv is useful, to pass the destination first.

If the file names have no whitespace nor any of \"', then you can simply provide the file names as input to xargs (the echo command is a bash builtin, so it isn't subject to the command line length limit; if you see !: event not found, you need to enable globbing syntax with shopt -s extglob):

echo !(*.jpg|*.png|*.bmp) | xargs mv -t targetdir

You can use the -0 option to xargs to use null-delimited input instead of the default quoted format.

printf '%s\0' !(*.jpg|*.png|*.bmp) | xargs -0 mv -t targetdir

Alternatively, you can generate the list of file names with find. To avoid recursing into subdirectories, use -type d -prune. Since no action is specified for the listed image files, only the other files are moved.

find . -name . -o -type d -prune -o \
       -name '*.jpg' -o -name '*.png' -o -name '*.bmp' -o \
       -exec mv -t targetdir/ {} +

(This includes dot files, unlike the shell wildcard methods.)

If you don't have GNU utilities, you can use an intermediate shell to get the arguments in the right order. This method works on all POSIX systems.

find . -name . -o -type d -prune -o \
       -name '*.jpg' -o -name '*.png' -o -name '*.bmp' -o \
       -exec sh -c 'mv "$@" "$0"' targetdir/ {} +

In zsh, you can load the mv builtin:

setopt extended_glob
zmodload zsh/files
mv -- ^*.(jpg|png|bmp) targetdir/

or if you prefer to let mv and other names keep referring to the external commands:

setopt extended_glob
zmodload -Fm zsh/files b:zf_\*
zf_mv -- ^*.(jpg|png|bmp) targetdir/

or with ksh-style globs:

setopt ksh_glob
zmodload -Fm zsh/files b:zf_\*
zf_mv -- !(*.jpg|*.png|*.bmp) targetdir/

Alternatively, using GNU mv and zargs:

autoload -U zargs
setopt extended_glob
zargs -- ./^*.(jpg|png|bmp) -- mv -t targetdir/

Best Answer

Related Solutions

ls – Argument List Too Long for ls Command

Bash Shell – Solving ‘mv: Argument List Too Long’

Related Question