How to Copy Files Transactionally in Linux

bashcplinux

I want to copy a file from A to B, which may be on different filesystems.

There are some additional requirements:

The copy is all or nothing, no partial or corrupt file B left in place on crash;
Do not overwrite an existing file B;
Do not compete with a concurrent execution of the same command, at most one can succeed.

I think this gets close:

cp A B.part && \
ln B B.part && \
rm B.part

But 3. is violated by the cp not failing if B.part exists (even with -n flag). Subsequently 1. could fail if the other process 'wins' the cp and the file linked into place is incomplete. B.part could also be an unrelated file, but I'm happy to fail without trying other hidden names in that case.

I think bash noclobber helps, does this work fully? Is there a way to get without the bash version requirement?

#!/usr/bin/env bash
set -o noclobber
cat A > B.part && \
ln B.part B && \
rm B.part

Followup, I know some file systems will fail at this anyway (NFS). Is there a way to detect such filesystems?

Some other related but not quite the same questions:

Approximating atomic move across file systems?

Is mv atomic on my fs?

is there a way to atomically move file and directory from tempfs to ext4 partition on eMMC

https://rcrowley.org/2010/01/06/things-unix-can-do-atomically.html

Best Answer

rsync does this job. A temporary file is O_EXCL created by default (only disabled if you use --inplace) and then renamed over the target file. Use --ignore-existing to not overwrite B if it exists.

In practice, I never experienced any problems with this on ext4, zfs or even NFS mounts.

Related Solutions

Shell – Make cp return an error value if the target exists

You could do:

(set -C &&  cat < /path/to/src > /path/to/dest)

It won't copy anything but the content of the file though (not the permissions, ownership or sparseness as some cp implementations do).

linux rename embedded ext4 tmpfs – How to Atomically Move File from tmpfs to ext4 on eMMC

Is moving file from tmpfs to ext4 atomic?

No. Renames as such only work within a filesystem. The manual page for rename(2) explicitly mentions the error that is returned if trying to rename across mount points:

EXDEV oldpath and newpath are not on the same mounted filesystem.

Moves across file systems need to be done as a combination of a copy and a delete. mv will do this for you if the rename() doesn't work, but it will not be atomic in that case.

The simple way to work around that would indeed be to first copy the file to a temporary location on the same filesystem. In general, it's simplest to place the temporary file in the same directory as the final destination, since that's the only place that's guaranteed to be on the same filesystem. Of course that requires that any process working on the files there will have some logic to ignore the temporary based on its name.

Roughly, something like this should work for one file:

cp /src/filename /dst/filename.tmp &&
mv /dst/filename.tmp /dst/filename &&
rm /src/filename

Note that the process you describe for a directory is essentially this:

cp -r /src/dir /dst/dir.tmp && 
mv /dst/dir /dst/dir.bak    &&
mv /dst/dir.tmp /dst/dir    &&
rm -r /dst/dir.bak

Which is not bad, but is not atomic. There's a moment of time between the two runs of mv (or calls to rename()), when /dst/dir does not exist. That could be worked around by accessing the directory through a symlink, since the link can be atomically replaced with a rename.

Best Answer

Related Solutions

Shell – Make cp return an error value if the target exists

linux rename embedded ext4 tmpfs – How to Atomically Move File from tmpfs to ext4 on eMMC

Related Question