How to modify a lower file through an OverlayFS directory

I ran into the exact same problem (trying to do the exact same thing: a read-only base system for a PXE boot environment).

The behavior I'm seeing exactly matches that in this Ubuntu bug report. I cannot write changes to existing files, but I can delete them (but they're preserved in the lower filesystem... with the upper layer just recording their absence) and then write them back again (at which point the upper layer has a copy which I can edit).

From what I can dig up, this is happening because OverlayFS and NFS don't play nice together. The thing that we're expecting to happen, when we try to modify a file from the lower filesystem, is called "copy up", and it breaks when OverlayFS tries to use a NFS filesystem because NFS doesn't support xattrs.

Thus far, the only promising avenue I can find is that it's possible to use fuse_xattrs to emulate xattrs on your NFS mount (described here), but it requires that you have two mounts of your NFS share: the "real" one, and the "xattr-enhanced" one that needs the first.

An attempt at an online solution, but its not quite there.

The setup (in e.g. /tmp directory, as root):

LOWER=$HOME
mkdir u1 w1 o1 O
mount -t overlay overlay -o lowerdir=$LOWER,upperdir=u1,workdir=w1 o1
mount --bind o1 O

Then you can work in O directory, which is an overlay over $LOWER. When you want to do the snapshot:

mkdir u2 w2 o2
mount -t overlay overlay -o lowerdir=o1,upperdir=u2,workdir=w2 o2

(Note that nested overlays like this won't work on older kernels).

But then I want some way to atomically change the bind mount at O to point to o2 instead of o1. I don't know how to do this other than:

umount O
mount --bind o2 O

(Not atomic; there is a window where O is unmounted).

Ideally, running processes could continue to run without knowing that the underlying filesystem of O had changed from o1 to o2. I don't know if this is possible, or whether changing the underlying filesystem of O like this will disrupt open applications too much. I need to investigate further.

Then, once O has been redirected to o2, we can remount o1 read-only as a precaution, then perform an offline merge using for example rdiffdir or overlayfs-tools.

Finally, we would want some way to atomically remount o2 as lowerdir=$HOME,upperdir=u2,workdir=w2 so that o1, u1 and w1 (all now empty dirs) could be removed. Again, I don't know if this is possible.

Otherwise, we can achieve snapshots by just nesting overlays deeper and deeper and leaving the overlay and upper dirs for each mounted without attempting to merge or cleanup. But there is probably a limit to the number of nested overlays that can be mounted. And at some point, we still need to merge the layers downwards if we want to persist changes.