Split and merge – with DVDs!

cat

Say I have a large file called foo.tar.xz.
I split the file into parts of just under 4.7GB each, using split -b 4689887232 foo.tar.xz foo.tar.xz., which gives me files named foo.tar.xz.aa, foo.tar.xz.ab, etc.
Then I write each file to a different DVD and send it to Alice using a rather large homing pigeon.

Now Alice could insert each DVD, copy each file to her PC, and then use cat and xz to get the contents of foo: cat foo.tar.xz.* | tar xfJ -

Now Assume that Alice has just enough space on her PC to store the extracted contents of foo. Is there some flavour of cat that will read these files directly from DVDs and pause the stream to allow her to insert the next DVD? Something like pausecat or volumecat?

Best Answer

I don't know of such a cat flavor, but here's a solution which almost works:

  1. In one terminal run mkfifo myfifo; tail -c +1 -f myfifo | tar xfJ -
  2. Insert first DVD into DVD-ROM drive (ex. dev/sr0).
  3. In a second terminal run dd if=/dev/sr0 of=myfifo
  4. When dd completes, remove the DVD.
  5. If you have another DVD, insert into the DVD-ROM drive and go to step 3, else go to step 6.
  6. In the first terminal, press CTRL-C to kill tail and tar.

How it works

The trick is to use tail to continuously read from a named pipe. The output of tail is then piped to the command you want to run. Initially the pipe is unopened so nothing happens. But when you begin pumping data into the pipe with dd, tail picks it up and forwards it to your command.

The magic happens when the flow of data to the pipe stops: tail keeps its standard out file descriptor open, which causes your command to pause. Meanwhile, tail simply waits for more input.

The problem

The reason I said it's almost working is because there seems to be a buffering issue which causes tail not to write the last bit of data that's fed into it. My hope is that someone can provide the insight to address this.

Related Question