Filesystems – How to Create Small Disk Image with Large Partitions

embeddedext3ext4filesystems

I am building a disk image for an embedded system (to be placed on an 4GB SD card). I want the system to have two partitions. A 'Root'(200Mb), and a 'Data' partition(800Mb).

I create an empty 1GB file with dd.
Then I use parted to set up the partitions.
I mount them each in a loop device then format them; ext2 for 'Root' ext4 for 'Data'. Add my root file system to the 'Root' partition and leave 'Data' empty.

Here's where the problem is. I am now stuck with a 1GB image, with only 200MB of data on it. Shouldn't I, in theory, be able to truncate the image down to say.. 201MB and still have the file system mountable? Unfortunately I have not found this to be the case.

I recall in the past having used a build environment from Freescale that used to create 30Mb images, that would have partitions for utilizing an entire 4GB sdcard. Unfortunately, at this time, I can not find how they were doing that.

I have read the on-disk format for the ext file system, and if there is no data in anything past the first super block (except for backup super blocks, and unused block tables) I thought I could truncate there.

Unfortunately, when I do this, the mounting system freaks out. I can then run FSCK, restore the super blocks, and block tables, and can mount it then no problem. I just don't think that should be necessary.

Perhaps a different file system could work? Any ideas?

thanks,

edit

changed partition to read file system. The partition is still there and deoesn't change, but the file system is getting destroyed after truncating the image.

edit

I have found the case to be that when I truncate the file to a size just larger than the first set of 'Data' partition superblock and inode/block tables, (Somewhere in the data-block range) the file system becomes umountable without doing a fsck to restore the rest of the super blocks and block/inode tables

Best Answer

Firstly, writing a sparse image to a disk will not result in anything but the whole of the size of that image file - holes and all - covering the disk. This is because handling of sparse files is a quality of the filesystem - and a raw device (such as the one to which you write the image) has no such thing yet. A sparse file can be stored safely and securely on a medium controlled by a filesystem which understands sparse files (such as an ext4 device) but as soon as you write it out it will envelop all that you intend it to. And so what you should do is either:

  1. Simply store it on an fs which understands sparse files until you are prepared to write it.

  2. Make it two layers deep...

    • Which is to say, write out your main image to a file, create another parent image with an fs which understands sparse files, then copy your image to the parent image, and...

    • When it comes time to write the image, first write your parent image, then write your main image.

Here's how to do 2:

  • Create a 1GB sparse file...

    dd bs=1kx1k seek=1k of=img </dev/null

  • Write two ext4 partitions to its partition table: 1 200MB, 2 800MB...

    printf '%b\n\n\n\n' n '+200M\nn\n' 'w\n\c' | fdisk img

  • Create two ext4 filesystems on a -Partitioned loop device and put a copy of the second on the first...

    sudo sh -c '
    for p in "$(losetup --show -Pf img)p"*        ### the for loop will iterate
    do    mkfs.ext4 "$p"                          ### over fdisks two partitions
          mkdir -p ./mnt/"${p##*/}"               ### and mkfs, then mount each
          mount "$p" ./mnt/"${p##*/}"             ### on dirs created for them
    done; sync; cd ./mnt/*/                       ### next we cp a sparse image
    cp --sparse=always "$p" ./part2               ### of part2 onto part1
    dd bs=1kx1k count=175 </dev/zero >./zero_fill ### fill out part1 w/ zeroes
    sync; cd ..; ls -Rhls .                       ### sync, and list contents
    umount */; losetup -d "${p%p*}"               ### last umount, destroy
    rm -rf loop*p[12]/ '                          ### loop devs and mount dirs

    mke2fs 1.42.12 (29-Aug-2014)
Discarding device blocks: done
Creating filesystem with 204800 1k blocks and 51200 inodes
Filesystem UUID: 2f8ae02f-4422-4456-9a8b-8056a40fab32
Superblock backups stored on blocks:
    8193, 24577, 40961, 57345, 73729

Allocating group tables: done
Writing inode tables: done
Creating journal (4096 blocks): done
Writing superblocks and filesystem accounting information: done

mke2fs 1.42.12 (29-Aug-2014)
Discarding device blocks: done
Creating filesystem with 210688 4k blocks and 52752 inodes
Filesystem UUID: fa14171c-f591-4067-a39a-e5d0dac1b806
Superblock backups stored on blocks:
    32768, 98304, 163840

Allocating group tables: done
Writing inode tables: done
Creating journal (4096 blocks): done
Writing superblocks and filesystem accounting information: done

175+0 records in
175+0 records out
183500800 bytes (184 MB) copied, 0.365576 s, 502 MB/s
./:
total 1.0K
1.0K drwxr-xr-x 3 root root 1.0K Jul 16 20:49 loop0p1
   0 drwxr-xr-x 2 root root   40 Jul 16 20:42 loop0p2

./loop0p1:
total 176M
 12K drwx------ 2 root root  12K Jul 16 20:49 lost+found
 79K -rw-r----- 1 root root 823M Jul 16 20:49 part2
176M -rw-r--r-- 1 root root 175M Jul 16 20:49 zero_fill

./loop0p1/lost+found:
total 0

./loop0p2:
total 0

  • Now that's a lot of output - mostly from mkfs.ext4 - but notice especially the ls bits at the bottom. ls -s will show the actual -size of a file on disk - and it is always displayed in the first column.

  • Now we can basically reduce our image to only the first partition...

    fdisk -l img

    Disk img: 1 GiB, 1073741824 bytes, 2097152 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xc455ed35

Device Boot  Start     End Sectors  Size Id Type
img1          2048  411647  409600  200M 83 Linux
img2        411648 2097151 1685504  823M 83 Linux

  • There fdisk tells us there are 411647 +1 512 byte sectors in the first partition of img...

    dd seek=411648 of=img </dev/null

  • That truncates the img file to only its first partition. See?

    ls -hls img

    181M -rw-r--r-- 1 mikeserv mikeserv 201M Jul 16 21:37 img

  • ...but we can still mount that partition...

    sudo mount "$(sudo losetup -Pf --show img)p"*1 ./mnt

  • ...and here are its contents...

    ls -hls ./mnt

    total 176M
 12K drwx------ 2 root root  12K Jul 16 21:34 lost+found
 79K -rw-r----- 1 root root 823M Jul 16 21:34 part2
176M -rw-r--r-- 1 root root 175M Jul 16 21:34 zero_fill

  • And we can append the stored image of the second partition to the first...

    sudo sh -c '
    dd seek=411648 if=./mnt/part2 of=img
    umount ./mnt; losetup -D
    mount "$(losetup -Pf --show img)p"*2 ./mnt
    ls ./mnt; umount ./mnt; losetup -D'

    1685504+0 records in
1685504+0 records out
862978048 bytes (863 MB) copied, 1.96805 s, 438 MB/s
lost+found

  • Now that has grown our img file: it's no longer sparse...

    ls -hls img

    1004M -rw-r--r-- 1 mikeserv mikeserv 1.0G Jul 16 21:58 img

  • ...but removing that is as simple the second time as it was the first, of course...

    dd seek=411648 of=img </dev/null
ls -hls img

    181M -rw-r--r-- 1 mikeserv mikeserv 201M Jul 16 22:01 img
Related Question