Bash – Fix Sed Command Argument List Too Long Error

bashsedshell-script

I'm trying to execute this command

sed -i -e "s/BASE_64/$BASE_64/" FILE_NAME

where $BASE_64 is a base 64 representation of a file content.

sed gives me an error since the string is too long.

Argument list too long

How is it possible to avoid this error?

Best Answer

First, save the base64-encoded data in a file called, e.g., base64.txt.

For example:

base64 < originalfile > base64.txt

Then:

printf '%s\n' '/BASE64/r base64.txt' 1 '/BASE64/d' w | ed FILENAME

This uses ed to search in FILENAME for a line containing the string BASE64, insert the contents of base64.txt after that line, go back to the first line, then search for the line with string BASE64 again and delete it. The w command in ed saves the modified file.

Related Solutions

Shell – /bin/cat: Argument list too long

If you want a line-count for each individual file:

find . -type f -exec wc -l {} + | awk '! /^[ 0-9]+[[:space:]]+total$/'

I've excluded the total lines because there will be several of them with this many files being processed. The find ... -exec ... + will try to fit as many filenames onto a single command line as possible, but that will be a LOT less than 119766 files....probably only several thousand (at most) per invocation of wc, and each one will result in its own independent 'total' line.

If you want the total number of lines in all files combined, here's one way of doing it:

find . -type f -exec wc -l {} + | 
    awk '/^[ 0-9]+[[:space:]]+total$/ {print $1}' | 
    xargs | sed -e 's/ /+/g' | bc

This prints only the line counts on the total lines, pipes that into xargs to get the counts all on one line, then sed to transform the spaces into + signs, and then pipes the lot into bc to do the calculation.

Example output:

$ cd /usr/share/doc
$ find . -type f -exec wc -l {} + | 
    awk '/^[ 0-9]+[[:space:]]+total$/ {print $1}' | 
    xargs | sed -e 's/ /+/g' | bc 
53358931

Fix ‘Argument List Too Long’ When Zipping Large Files

extract from man zip ( linux version )

   zip -@ foo
   will store the files listed one per line on stdin in foo.zip.

example from the same man page

   find . -name "*.[ch]" -print | zip source -@

So steps will be :

build a list off all files to be archive , format must one file name by line
run zip command

cat BIG_FILENAME_LIST.txt | zip thebigziparchive -@

Best Answer

Related Solutions

Shell – /bin/cat: Argument list too long

Fix ‘Argument List Too Long’ When Zipping Large Files

Related Question