Linux Filesystems – Mountable Filesystem that Automatically Splits Files

filesystemslinux

Is there a [virtual-]filesystem that can automatically split files in storage but present them to the user as single files?

For example:

The user sees: /user/files/10TBfile
But it is stored as: /backingstorage/user/files/10TBfile.{1..100}

Basically the same way a split archive works but in real time.
I feel like it should be possible since it's basically storing each virtual disk block as a separate file but I don't know of any existing solutions for it.

If you're curious, the end goal might be something similar to this question: Divide out local files in different servers with limited space with rsync except that I have single large files that need to be split and the files need to be updated in real-time so a daily cron/rsync and split tar are out of the question. I already have the remote drives mounted so I just need a way to split the file and present it as a single file to the user.

Thanks!

Best Answer

What you want is chunkfs:

ChunkFS is a FUSE based filesystem that allows you to mount an arbitrary file or block device as a directory tree of files that each represent a chunk of user-specified size of the mounted file.

It was written for the same purpose as yours:

ChunkFS was originally written for making space-efficient incremental backups of encrypted filesystem images using rsync. Using the --link-dest option of rsync, you can create incremental backups from the ChunkFS-mounted image where any chunk that hasn't changed since the last backup will be a hard link to the corresponding chunk from the previous backup.

Related Solutions

Linux – Browsable compressed filesystem archive with full attributes

Do you know about tar's t function for printing out the contents of the archive? You can use it on a gzip compressed archive like this:

tar ztvf archive.tgz

And it will print out the files in a long list like format including timestamps and sizes. It turns out that if you use two v options when creating the archive that it will long list out the files (instead of just plain listing them) as it adds them to the archive, so you can do something like this to auto create an index while you're backing up in one go:

tar cvvpzf archive.tgz /home/foo/bar > archive.tgz.index

Note the use of two v options. Many other commands allow you to increase verbosity this way (ssh for example).

Linux Rsync – How to Sync to/from Tar Archive

I haven't actually tried this, but it should work.

Using 'archivemount' (source from:)

http://www.cybernoia.de/software/archivemount/

and a 'libarchive' included in many distros (suse, redhat, etc)...

Or a pre-built one from:

https://rpmfind.net/linux/rpm2html/search.php?query=archivemount

You can mount a tar-archive using the fusermount facility in linux.

From there, you should be able to use rsync directly to the final system.

I wrote a simple passthrough batchfile to test rsync's passthrough:

#!/bin/bash
# ussh -- use root@ssh to target system
exec ssh  root@"$@"

then, as a test, used rsync to pass dir 'test1' to 'ishtar', calling it /tmp/test2 on the target:

RSYNC_RSH=$PWD/Ussh rsync -uva /tmp/test1/ ishtar:/tmp/test2

It will ask you for the password of the target sys's root logon, or you could setup the target system to accept a root login via a certificate so no password would be needed.

This would seem to be the most efficient way to do what you want (you might need to modify the rsync options to not copy dir times and things like that), but is this the type of thing you were looking for?

-Astara

Best Answer

Related Solutions

Linux – Browsable compressed filesystem archive with full attributes

Linux Rsync – How to Sync to/from Tar Archive

Related Question