Fix ‘Argument List Too Long’ When Zipping Large Files

bash

I'm trying to zip a certain large number of files by creating an array of them. Then I tried to run the zip command but met with "Argument list too long" error.

declare -a arr=()       

fixed=5
for i in `seq 10 1 200`; do
    for j in `seq $((i+fixed)) 1 200`; do
        arr+=("${i}_${j}.xxx")
    done
done

new_arr=$(printf ",%s" "${arr[@]}")
new_arr=${new_arr:1}

zip all_data.zip {$new_arr}

Best Answer

extract from man zip ( linux version )

   zip -@ foo
   will store the files listed one per line on stdin in foo.zip.

example from the same man page

   find . -name "*.[ch]" -print | zip source -@

So steps will be :

build a list off all files to be archive , format must one file name by line
run zip command

cat BIG_FILENAME_LIST.txt | zip thebigziparchive -@

Related Solutions

Bash Arrays – Using Variables in Array Names

Some ideas:

A "parameter expansion" of a variable value (the ${...} part):
```
echo "arr$COUNTER[0] = ${arr$COUNTER[0]}"
```
will not work. You may get around by using eval (but I do not recommend it):
```
eval echo "arr$COUNTER[0] = \${arr$COUNTER[0]}"
```
That line could be written as this:
```
i="arr$COUNTER[0]"; echo "$i = ${!i}"
```
That is called indirection (the !) in Bash.
A similar issue happens with this line:
```
declare -a "arr$COUNTER=($field)"
```
Which should be split into two lines, and eval used:
```
declare -a "arr$COUNTER"
eval arr$COUNTER\=$ \$field $
```
Again, I do not recommend using eval (in this case).
As you are reading the whole file into the memory of the shell, we may as well use a simpler method to get all lines into an array:
```
readarray -t lines <"VALUES_FILE.txt"
```
That should be faster than calling awk for each line.

An script with all the above could be:

#!/bin/bash
valfile="VALUES_FILE.txt"

readarray -t lines <"$valfile"             ### read all lines in.

line_count="${#lines[@]}"
echo "Total number of lines $line_count"

for ((l=0;l<$line_count;l++)); do
    echo "Counter value is $l"             ### In which line are we?
    echo "Field = ${lines[l]}"             ### kepth only to help understanding.
    k="arr$l"                              ### Build the variable arr$COUNTER
    IFS=" " read -ra $k <<<"${lines[l]}"   ### Split into an array a line.
    eval max=\${#$k[@]}                    ### How many elements arr$COUNTER has?
    #echo "field $field and k=$k max=$max" ### Un-quote to "see" inside.
    for ((j=0;j<$max;j++)); do             ### for each element in the line.
        i="$k[$j]"; echo "$i = ${!i}"      ### echo it's value.
    done
done
echo "The End"
echo

However, still, AWK may be faster, if we could execute what you need in AWK.

A similar processing could be done in awk. Assuming the 6 values will be used as an IP (4 of them) and the other two are a number and an epoch time.

Just a very simple sample of an AWK script:

#!/bin/sh
valfile="VALUES_FILE.txt"
awk '
NF==6 { printf ( "IP: %s.%s.%s.%s\t",$1,$2,$3,$4)
        printf ( "number: %s\t",$5+2)
        printf ( "epoch: %s\t",$6)
        printf ( "\n" )
    }
' "$valfile"

Just make a new question with the details.

Bash – Fix Sed Command Argument List Too Long Error

First, save the base64-encoded data in a file called, e.g., base64.txt.

For example:

base64 < originalfile > base64.txt

Then:

printf '%s\n' '/BASE64/r base64.txt' 1 '/BASE64/d' w | ed FILENAME

This uses ed to search in FILENAME for a line containing the string BASE64, insert the contents of base64.txt after that line, go back to the first line, then search for the line with string BASE64 again and delete it. The w command in ed saves the modified file.

Best Answer

Related Solutions

Bash Arrays – Using Variables in Array Names

Bash – Fix Sed Command Argument List Too Long Error

Related Question