So en route from my old laptop to a new one my old laptop's hard drive got some physical damage. badblocks reports 64 bad sectors. I had a two-month-old Ubuntu GNOME setup with a split / and /home partitions. From what I can tell, a few sectors in / were damaged, but that's not an issue. On the other hand, /home's partition gives me this annotated ddrescue log:
# Rescue Logfile. Created by GNU ddrescue version 1.17
# Command line: ddrescue -d -r -1 /dev/sdb2 home.img home.log
# current_pos current_status
0x6788008400 -
# pos size status
0x00000000 0x6788000000 +
0x6788000000 0x0000A000 -
first 10 sectors of the ext4 journal
0x678800A000 0x2378016000 +
0x8B00020000 0x00001000 -
inode table entries for /pietro (my $HOME) and a few folders within
0x8B00021000 0x00006000 +
0x8B00027000 0x00001000 -
unknown (inode table?)
0x8B00028000 0x00004000 +
0x8B0002C000 0x00001000 -
unknown (inode table?)
0x8B0002D000 0x001DC000 +
0x8B00209000 0x00001000 -
unknown (inode table?)
0x8B0020A000 0x00090000 +
0x8B0029A000 0x00001000 -
unknown (inode table?)
0x8B0029B000 0x4420E65000 +
I made the annotations with use of debugfs's icheck and testb commands; all the damaged blocks are marked used. Some superblock stats:
Block size: 4096
Fragment size: 4096
Reserved GDT blocks: 972
Blocks per group: 32768
Fragments per group: 32768
Inodes per group: 8192
Inode blocks per group: 512
So my questions are:
- Can I find out exactly what those five unknown blocks were, if not inode entries? My suspicion is that they are inode table entries, but
icheckdoesn't want to say. If they are, can I find out which inodes? - Can I still recover these inode table entries from the journal by hand, even though the first 10 blocks of the journal are lost?
I'd rather not do this data recovery with fsck, which will just dump all my files i n /lost+found in a giant mess of flattened directory structure and duplicate files…
Thanks.
Best Answer
All right, so for the first question it turns out the
debugfsstatscommand tells what the starting blocks for every section of a group are. In addition, I guessed that inumbers had to be consecutive and increasing, so basic addition of the offset into the inode table and theimapcommand gave me the first inumbers; it also confirmed my suspicion about the last bad sector, where my block group calculations indicated it was in the wrong group.Since a block is 4096 bytes and each inode table entry is 256 bytes, there are 16 inodes per block. So I now have all 80 lost inode table entries by inumber.
Now let's turn to the journal. I wrote a small tool that dumps information in each block of the journal. Since the journal superblock was missing, there were two pieces of information that I needed for this that were lost:
Fortunately, if I forced one (or both) of these switches on, some of the descriptor blocks in the journal overflowed its block, proving that those flags were not set.
One awk script (
fulllog.awk) later, I have a log of the formWith this, another awk script (
dumpallfor.awk) dumps all the blocks:So that last block is truly lost :( With any luck I can find out what files were there with
debugfs'sncheckcommand.So I have a bunch of blocks. And they all appear to differ! Now what?
I could go by the revocation records, but I can't seem to parse that structure meaningfully. I could go by the commit record timestamps, but before I try that, I want to see just how each inode table block differs. So I wrote another quick program (
diff.go) to find that out.For the most part, files that do differ differ only in timestamps, so we can just choose the file with the latest timestamps. We'll do that later. For all other files, we get this:
Hm, that's not good... The file with differing size will be a problem, and I have no idea what to do about the two OSD1 files. I also tried using
debugfs'sncheckto see what the files were, but we don't have a match.I then found out which block dumps have the latest timestamps for now (same repo,
latest.go). The important thing to note is that I had the blocks scanned in chronological order by commit time. This is not necessarily the same as numerical order by block number; the journal is not always stored in chronologically increasing order.As it turns out, however, the newest block (by commit time) is indeed the one with the latest timestamps!
Let's try these latest blocks and see if we can recover anything from them.
After that my home directory is back!
Now let's find out what those three differing files were...
The only important thing there is Liferea's configuration directory, but I don't think that was corrupted; it was one of the OSD1-differing ones.
And let's find out about those 16 inodes in the final block, the one that we could not recover:
In short:
So not casualtyless but not a total loss, and I learned more about ext4 in the process. Thanks anyway!
UPDATE
Might as well put this out there:
And in case I'm not weird enough, the downloaded pictures were:
sOMe4P7.jpg(a parody of the Law & Order title card with "& KNUCKLES" added to it)tumblr_nfjvg292T21s4pk45o1_1280.png(screenshot of this tweet from J. K. Rowling)tumblr_mrfex1kuxa1sbx6kgo1_500.jpg(picture of a "Windows did not shut down successfully." error message on a billboard at what appears to be some sporting event)1*-vuzP4JAoPf9S6ZdHNR_Jg.jpeg(this comic)These were all shared by friends in chats.
I guess I'll keep this updated? (Not like it would make a difference...) I know I can recover everything; the only question is when =P