Maximum Value for the bs Argument of dd

dd

I want to write a CGI, which must read a specified number of bytes from STDIN. My idea is to do it this way:

dd bs=$CONTENT_LENGTH count=1

But I was wondering, if the block size is limited by anything else but the RAM.

$ dd bs=1000000000000
dd: memory exhausted by input buffer of size 1000000000000 bytes (931 GiB)

The manual page of GNU's coreutils does not specify any limit.

Best Answer

The POSIX specifications for dd don’t specify a maximum explicitly, but there are some limits:

On a 64-bit platform, size_t is 64 bits in length; in addition, it’s unsigned, so dd will fail when given values greater than 264 – 1:

$ dd if=/dev/zero of=/dev/null bs=18446744073709551616
dd: invalid number: ‘18446744073709551616’

On Linux on 64-bit x86, SSIZE_MAX is 0x7fffffffffffffffL (run echo SSIZE_MAX | gcc -include limits.h -E - to check), and that’s the input limit:

$ dd if=/dev/zero of=/dev/null bs=9223372036854775808
dd: invalid number: ‘9223372036854775808’: Value too large for defined data type

$ dd if=/dev/zero of=/dev/null bs=9223372036854775807
dd: memory exhausted by input buffer of size 9223372036854775807 bytes (8.0 EiB)

Once you find a value which is accepted, the next limit is the amount of memory which can be allocated, since dd needs to allocate a buffer before it can read into it.

Once you find a value which can be allocated, you’ll hit the read limit (on Linux and other systems with similar limits), unless you use GNU dd and specify iflag=fullblock:

$ dd if=/dev/zero of=ddtest bs=4294967296 count=1
0+1 records in
0+1 records out
2147479552 bytes (2.1 GB, 2.0 GiB) copied, 38.3037 s, 56.1 MB/s

(dd copied just under 231 bytes, i.e. the Linux limit mentioned above, not even half of what I asked for).

As explained in the Q&A linked above, you’ll need fullblock to reliably copy all the input data in any case, for any value of bs greater than 1.

Related Question