Linux – Wrong inode: corrupted f2fs, fsck.f2fs cannot repair

Firstly, writing a sparse image to a disk will not result in anything but the whole of the size of that image file - holes and all - covering the disk. This is because handling of sparse files is a quality of the filesystem - and a raw device (such as the one to which you write the image) has no such thing yet. A sparse file can be stored safely and securely on a medium controlled by a filesystem which understands sparse files (such as an ext4 device) but as soon as you write it out it will envelop all that you intend it to. And so what you should do is either:

1. Simply store it on an fs which understands sparse files until you are prepared to write it.

2. Make it two layers deep...

   • Which is to say, write out your main image to a file, create another parent image with an fs which understands sparse files, then copy your image to the parent image, and...

   • When it comes time to write the image, first write your parent image, then write your main image.

Here's how to do 2:

• Create a 1GB sparse file...

  dd bs=1kx1k seek=1k of=img </dev/null
  

• Write two ext4 partitions to its partition table: 1 200MB, 2 800MB...

  printf '%b\n\n\n\n' n '+200M\nn\n' 'w\n\c' | fdisk img
  

• Create two ext4 filesystems on a -Partitioned loop device and put a copy of the second on the first...

  sudo sh -c '
      for p in "$(losetup --show -Pf img)p"*        ### the for loop will iterate
      do    mkfs.ext4 "$p"                          ### over fdisks two partitions
            mkdir -p ./mnt/"${p##*/}"               ### and mkfs, then mount each
            mount "$p" ./mnt/"${p##*/}"             ### on dirs created for them
      done; sync; cd ./mnt/*/                       ### next we cp a sparse image
      cp --sparse=always "$p" ./part2               ### of part2 onto part1
      dd bs=1kx1k count=175 </dev/zero >./zero_fill ### fill out part1 w/ zeroes
      sync; cd ..; ls -Rhls .                       ### sync, and list contents
      umount */; losetup -d "${p%p*}"               ### last umount, destroy
      rm -rf loop*p[12]/ '                          ### loop devs and mount dirs
  

  ---

  mke2fs 1.42.12 (29-Aug-2014)
  Discarding device blocks: done
  Creating filesystem with 204800 1k blocks and 51200 inodes
  Filesystem UUID: 2f8ae02f-4422-4456-9a8b-8056a40fab32
  Superblock backups stored on blocks:
      8193, 24577, 40961, 57345, 73729
  
  Allocating group tables: done
  Writing inode tables: done
  Creating journal (4096 blocks): done
  Writing superblocks and filesystem accounting information: done
  
  mke2fs 1.42.12 (29-Aug-2014)
  Discarding device blocks: done
  Creating filesystem with 210688 4k blocks and 52752 inodes
  Filesystem UUID: fa14171c-f591-4067-a39a-e5d0dac1b806
  Superblock backups stored on blocks:
      32768, 98304, 163840
  
  Allocating group tables: done
  Writing inode tables: done
  Creating journal (4096 blocks): done
  Writing superblocks and filesystem accounting information: done
  
  175+0 records in
  175+0 records out
  183500800 bytes (184 MB) copied, 0.365576 s, 502 MB/s
  ./:
  total 1.0K
  1.0K drwxr-xr-x 3 root root 1.0K Jul 16 20:49 loop0p1
     0 drwxr-xr-x 2 root root   40 Jul 16 20:42 loop0p2
  
  ./loop0p1:
  total 176M
   12K drwx------ 2 root root  12K Jul 16 20:49 lost+found
   79K -rw-r----- 1 root root 823M Jul 16 20:49 part2
  176M -rw-r--r-- 1 root root 175M Jul 16 20:49 zero_fill
  
  ./loop0p1/lost+found:
  total 0
  
  ./loop0p2:
  total 0
  

• Now that's a lot of output - mostly from mkfs.ext4 - but notice especially the ls bits at the bottom. ls -s will show the actual -size of a file on disk - and it is always displayed in the first column.

• Now we can basically reduce our image to only the first partition...

  fdisk -l img
  

  ---

  Disk img: 1 GiB, 1073741824 bytes, 2097152 sectors
  Units: sectors of 1 * 512 = 512 bytes
  Sector size (logical/physical): 512 bytes / 512 bytes
  I/O size (minimum/optimal): 512 bytes / 512 bytes
  Disklabel type: dos
  Disk identifier: 0xc455ed35
  
  Device Boot  Start     End Sectors  Size Id Type
  img1          2048  411647  409600  200M 83 Linux
  img2        411648 2097151 1685504  823M 83 Linux
  

• There fdisk tells us there are 411647 +1 512 byte sectors in the first partition of img...

  dd seek=411648 of=img </dev/null
  

• That truncates the img file to only its first partition. See?

  ls -hls img
  

  ---

  181M -rw-r--r-- 1 mikeserv mikeserv 201M Jul 16 21:37 img
  

• ...but we can still mount that partition...

  sudo mount "$(sudo losetup -Pf --show img)p"*1 ./mnt
  

• ...and here are its contents...

  ls -hls ./mnt
  

  ---

  total 176M
   12K drwx------ 2 root root  12K Jul 16 21:34 lost+found
   79K -rw-r----- 1 root root 823M Jul 16 21:34 part2
  176M -rw-r--r-- 1 root root 175M Jul 16 21:34 zero_fill
  

• And we can append the stored image of the second partition to the first...

  sudo sh -c '
      dd seek=411648 if=./mnt/part2 of=img
      umount ./mnt; losetup -D
      mount "$(losetup -Pf --show img)p"*2 ./mnt
      ls ./mnt; umount ./mnt; losetup -D'
  

  ---

  1685504+0 records in
  1685504+0 records out
  862978048 bytes (863 MB) copied, 1.96805 s, 438 MB/s
  lost+found
  

• Now that has grown our img file: it's no longer sparse...

  ls -hls img
  

  ---

  1004M -rw-r--r-- 1 mikeserv mikeserv 1.0G Jul 16 21:58 img
  

• ...but removing that is as simple the second time as it was the first, of course...

  dd seek=411648 of=img </dev/null
  ls -hls img
  

  ---

  181M -rw-r--r-- 1 mikeserv mikeserv 201M Jul 16 22:01 img
  

I think the reason you're confused is because you don't know what a directory is. To do this lets take a step back and examine how Unix filesystems work.

The Unix filesystem has several separate notions for addressing data on disk:

• data blocks are a group of blocks on a disk which have the contents of a file.
• inodes are special blocks on a filesystem, with a numerical address unique within that filesystem, which contains metadata about a file such as:
  • permissions
  • access / modification times
  • size
  • pointers to the data blocks (could be a list of blocks, extents, etc)
• filenames are hierarchical locations on a filesystem root that are mapped to inodes.

In other words, a "file" is actually composed of three different things:

1. a PATH in the filesystem
2. an inode with metadata
3. data blocks pointed to by the inode

Most of the time, users imagine a file to be synonymous to "the entity associated with the filename" - it's only when you're dealing with low-level entities or the file/socket API that you think of inodes or data blocks. Directories are one of those low-level entities.

You might think that a directory is a file that contains a bunch of other files. That's only half-correct. A directory is a file that maps filenames to inode numbers. It doesn't "contain" files, but pointers to filenames. Think of it like a text file that contains entries like this:

• . - inode 1234
• .. - inode 200
• Documents - inode 2008
• README.txt - inode 2009

The entries above are called directory entries. They are basically mappings from filenames to inode numbers. A directory is a special file that contains directory entries.

That's a simplification of course, but it explains the basic idea and other directory weirdness.

• Why don't directories know their own size?
  • Because they only contain pointers to other stuff, you have to iterate over their contents to find the size
• Why aren't directories ever empty?
  • Because they contain at least the . and .. entries. Thus, a proper directory will be at least as small as the smallest filesize that can contain those entries. In most filesystems, 4096 bytes is the smallest.
• Why is it that you need write permission on the parent directory when renaming a file?
  • Because you're not just changing the file, you're changing the directory entry pointing to the file.
• Why does ls show a weird number of "links" to a directory?
  • a directory can be referenced (linked to) by itself, its parent, its children.
• What does a hard link do and how does it differ from a symlink?
  • a hard link adds a directory entry pointing to the same inode number. Because it points to an inode number, it can only point to files in the same filesystem (inodes are local to a filesystem)
  • a symlink adds a new inode which points to a separate filename. Because it refers to a filename it can point to arbitrary files in the tree.

But wait! Weird things are happening!

ls -ld somedirectory always shows the filesize to be 4096, whereas ls -l somefile shows the actual size of a file. Why?

Point of confusion 1: when we say "size" we can be referring to two things:

• filesize, which is a number stored in the inode; and
• allocated size, which is the number of blocks associated with the inode times the size of each block.

In general, these are not the same number. Try running stat on a regular file and you'll see this difference.

When a filesystem creates a non-empty file, it usually eagerly allocates data blocks in groups. This is because files have a tendency to grow and shrink arbitrarily fast. If the filesystem only allocated as many data blocks as needed to represent the file, growing / shrinking would be slower, and fragmentation would be a serious concern. So in practice, filesystems don't have to keep reallocating space for small changes. This means that there may be a lot of space on disk that is "claimed" by files but completely unused.

What does the filesystem do with all this unused space? Nothing. Until it feels like it needs to. If your filesystem optimizer tool - maybe an online optimizer running in the background, maybe part of your fsck, maybe built-in to your filesystem itself - feels like it, it may reassign the data blocks of your files - moving used blocks, freeing unused blocks, etc.

So now we come to the difference between regular files and directories: because directories form the "backbone" of your filesystem, you expect that they may need to be accessed or modified frequently and should thus be optimized. And so you don't want them fragmented at all. When directories are created, they always max out all their data blocks in size, even when they only have so many directory entries. This is okay for directories, because, unlike files, directories are typically limited in size and growth rate.

The 4096 reported size of directories is the "filesize" number stored in the directory inode, not the number of entries in the directory. It isn't a fixed number - it's the maximum bytes that will fit into the allocated number of blocks for the directory. Typically, this is 512 bytes/block times 8 blocks allocated for a file with any contents - incidentally, for directories, the filesize and the allocated size are the same. Because it's allocated as a single group, the filesystem optimizer won't move its blocks around.

As the directory grows, more data blocks are assigned to it, and it will also max out those blocks by adjusting the filesize accordingly.

And so ls and stat will show the filesize field of the directory's inode, which is set to the size of the data blocks assigned to it.