Dd, seek with one block size, write with another block size

dddisksd card

Background Info:

Copying some .bin files to an SD card (to be read by an embedded device, no filesystem)
Commissioning the card requires some segments to be wiped (i.e. zero'd), and others to have binary files copied to them
Calling dd from a python script using subprocess module (as the dd operations involved are triggered by a sort of configuration script that needs to be parsed and validated first, I also make the user confirm the operation, as they might wipe out an important disk that is mistaken for the SD card)

Problem:

Writes to the SD card are slow with bs=512. For large spans, bs=8M is much faster.

Is it possible to somehow 'bs=512 seek={n_small_blocks}' and then change to 'bs=8M' for the actual write (once I've seek'd to the correct position)?

I found the following resource:
http://www.delorie.com/gnu/docs/textutils/coreutils_65.html

But it's not clear to me why 2 invocations are required, and how they're working together to accomplish what the guide claims they will.

UPDATE

Found the answer here:

https://superuser.com/questions/380717/how-to-output-file-from-the-specified-offset-but-not-dd-bs-1-skip-n

See my full solution below

Best Answer

Solution:

dd if='input_file.bin'           \
   of='/dev/sd{X}'               \
   bs={desired write block size} \
   seek={start offset in bytes}  \
   count={write size in bytes}   \
   oflag=seek_bytes              \
   iflag=count_bytes

From the man page:

count_bytes
    treat 'count=N' as a byte count (iflag only)

...

seek_bytes
    treat 'seek=N' as a byte count (oflag only)

This does seem to slow down the transfer a bit, but at least puts it in MB/s, instead of kB/s. Also, be sure to check the man page on your system, as it seems the ones available on the web (i.e. googling 'man dd') don't include these options.

Related Solutions

Files – How to Write One File into Another Using dd

To overwrite the start of the destination file without truncating it, give the notrunc conversion directive:

$ dd if=out/one.img of=out/go.img conv=notrunc

If you wanted the source file's data appended to the destination, you can do that with the seek directive:

$ dd if=out/one.img of=out/go.img bs=1k seek=9

This tells dd that the block size is 1 kiB, so that the seek goes forward by 9 kiB before doing the write.

You can also combine the two forms. For example, to overwrite the second 1 kiB block in the file with a 1 kiB source:

$ dd if=out/one.img of=out/go.img bs=1k seek=9 conv=notrunc

That is, it skips the first 1 kiB of the output file, overwrites data it finds there with data from the input file, then closes the output without truncating it first.

Understanding sparse files, dd, seek, inode block structure

Some quick answers: first, you didn't create a sparse file. Try these extra commands

dd if=/tmp/BIL of=/tmp/sparse seek=1000
ls -ls /tmp/sparse

You will see the size is 512003 bytes, but only takes 8 blocks. The null bytes have to occupy a whole block, and be on a block boundary for them to be possibly sparse in the filesystem.

Why does the second occurrence of "BIL" appear out of order?

because you are on a little-endian system and you are writing output in shorts. Use bytes, like cat does.
How does cat and other tools know to print in the correct order?

they work on bytes.
How do programs like ls discern between the "alleged" size and the allocated size?

ls and so on use the stat(2) system call which returns 2 values:
```
st_size;             /* total size, in bytes */ 
blkcnt_t  st_blocks; /* number of 512B blocks allocated */
```
What tools can I use to interrogate inode information?

stat is good.

Is there a tool where I can walk the direct and indirect blocks?

On ext2/3/4 you can use hdparm --fibmap with the filename:

$ sudo hdparm --fibmap ~/sparse 
filesystem blocksize 4096, begins at LBA 25167872; assuming 512 byte sectors.
byte_offset  begin_LBA    end_LBA    sectors
     512000  226080744  226080751          8

You can also use debugfs:

$ sudo debugfs /dev/sda3
debugfs:  stat <1040667>
Inode: 1040667   Type: regular    Mode:  0644   Flags: 0x0
Generation: 1161905167    Version: 0x00000000
User:   127   Group:   500   Size: 335360
File ACL: 0    Directory ACL: 0
Links: 1   Blockcount: 664
Fragment:  Address: 0    Number: 0    Size: 0
ctime: 0x4dd61e6c -- Fri May 20 09:55:24 2011
atime: 0x4dd61e29 -- Fri May 20 09:54:17 2011
mtime: 0x4dd61e6c -- Fri May 20 09:55:24 2011
Size of extra inode fields: 4
BLOCKS:
(0-11):4182714-4182725, (IND):4182726, (12-81):4182727-4182796
TOTAL: 83

Why does dd truncate my file and can dd or another tool write into the middle of a file?

Yes, dd can write into the middle. Add conv=notrunc.
Are there mechanisms to prevent sparse files be shrunk/grown? And if not, why are sparse files useful?

No. Because they take less space.

The sparse aspect of a file should be totally transparent to a program, which sometimes means the sparseness may be lost when the program updates a file.

Some copying utilities have options to preserve sparseness, eg tar --sparse, rsync --sparse.

Note, you can explicitly convert the suitably aligned zero blocks in a file to sparseness by using cp --sparse=always and the reverse, converting sparse space into real zeros, with cp --sparse=never.

Best Answer

Related Solutions

Files – How to Write One File into Another Using dd

Understanding sparse files, dd, seek, inode block structure

Related Question