Linux – quick way to find large stretches of zeros in files

command linedata-recoverylinux

I recently partially recovered a 2.5TB faulty disc. ddrescue created an image, which I can mount in loopback mode, 2.1TB are recovered, 450GB are missing, unfortunately spread all over the disk.

To see which files are affected, I could use filefrag -v and look at the map file generated by ddrescue.

BUT that would take ages. I found that since it’s only video files I’m recovering, large stretches of zeros are not to be expected, but they are present, where ddrescue didn’t read data from the disk.

So I would need a command to scan a file if there is an (arbitrary) large patch of all zeros in the file. In reality, these would be always a multiple of 512 bytes, and always begin at a 512 byte address. Is there a command that can scan a file for such a binary byte sequence (i.e. 512× '\0')?

Best Answer

I've modified xenoid's answer to look specifically for null bytes, based on this other question's answer about how to grep for null bytes:

grep -Pal '\x00{512}' the_files

Related Solutions

Data recovery on a corrupted 3TB disk

I've used photorec to recover files from a drive that windows saw as unformatted.
Unfortunately it doesn't satisfy your top 2 priorities, but it satisfies the rest of the priorities.

It will require a 2nd disk to store all of your recovered data on as well as it doesn't modify the original disk.

Linux – Rebuild massively fragmented files with a partial image and a list of their sectors

So I did run those ddrescue scripts (first made them executable with the “chmod +x” command, then called them with ./name_of_the_script) :

– At first the commands didn't work, ddrescue gave only errors, I had to edit the scripts again so that the parameters would be placed before the names of the input and output files. The commands then looked like this :

ddrescue -P -i 2115843346432 -s 17563648 -o 0  ST3000DM001-2.dd 201707222358.mp4
ddrescue -P -i 2115861041152 -s 65536 -o 17563648  ST3000DM001-2.dd 201707222358.mp4
ddrescue -P -i 2115861172224 -s 262144 -o 17629184  ST3000DM001-2.dd 201707222358.mp4
ddrescue -P -i 2115861499904 -s 65536 -o 17891328  ST3000DM001-2.dd 201707222358.mp4
ddrescue -P -i 2115861630976 -s 196608 -o 17956864  ST3000DM001-2.dd 201707222358.mp4
ddrescue -P -i 2115862024192 -s 131072 -o 18153472  ST3000DM001-2.dd 201707222358.mp4
...
ddrescue -P -i 2327182266368 -s 36864 -o 883752960  ST3000DM001-2.dd 201707222358.mp4

(Total size of that file : 883787365, or 883789824 with the slack space.)
(“-P” stands for “preview”, “-i” for “input position”, “-s” for “size”, “-o” for “output position”.)
(The paths could be omitted as the scripts, the image file and the expected output files were all in the same directory.)

– Then the first attempt produced an unreadable file, without a correct MP4 header. Why ? Because the list provided by Hard Disk Sentinel gives the physical/absolute sector numbers, but the logical cluster numbers (I verified by opening the image file with WinHex), so I had to add 264192x512 to the input offset calculation (the partition offset being 264192 sectors, or 129MB).

– Then it worked. It took just a few minutes and produced five video files, which are mostly readable, skippable to the end, with their expected content — I haven't watched them completely, but it seems as flawless as can be.

(I made all this on a secondary computer running on Knoppix live from a memory card, and used TeamViewer to command it from my primary computer on Windows 7, and also to be able to transfer the script files easily. Maybe there's a simpler setup for such purposes, but, well, it works ! :^p)

– But of course there are corrupted parts, since there were unreadable sectors in that partial image. How could I know where, quickly and reliably ? Well...
I had the idea to use ddrescue's “generate” mode, which creates logfiles (or mapfiles as they're called now) by parsing the output and considering that totally empty sectors are unread sectors, marked “?”, the rest being marked “+”. Since ddrescue expects an input file and an output file, but only the output file is actually parsed in that mode, I created dummy input files with this command, which copies only 1MB but extends the size to the size of the output files (just to save time and space) :

ddrescue -s 1048576 -x 883789824 201707222358.mp4 201707222358copy.mp4

Then I ran the “generate” command :

ddrescue -G 201707222358copy.mp4 201707222358.mp4 201707222358-generate.log

And then I opened those files with ddrescueview :

(Three of the six files are seriously damaged like the first one above, with large chunks of empty data, the three others only have a few corrupted sectors like the second one. The second one is the one which was not fragmented, I extracted it with a single ddrescue command.)

And then I patted myself on the back with one hand, while I was slapping my face with the other for having used that 3TB HDD every day for months with no backup... (At first it was supposed to store only temporary stuff, while I would be making room on other HDDs, but it took longer than anticipated, and I ran out of space to store such videos, and even my personal pictures and videos at some point, it could have been a major disaster, but “it's only a glitch”, as Dick Jones would have said.)

Best Answer

Related Solutions

Data recovery on a corrupted 3TB disk

Linux – Rebuild massively fragmented files with a partial image and a list of their sectors

Related Question