Linux – NFS COMMIT operation not being invoked despite fsync being used

linuxnfsrhel

I have been doing some elaborate testing to determine if our NetApp NFS (v3 protocol) server in async mode properly honors the client's fsync request. I've hit a wall when I discovered that Linux (RHEL 6, kernel 2.6.32-431.5.1) never issues a COMMIT op at all!!! That fact was uncovered by use of both the nfsstat tool, and by the nfstrace tool. Not a single COMMIT.

This seems to violate NFS semantics:

Version 3 clients use COMMIT operations when flushing safe asynchronous writes to the server during a close(2) or fsync(2) system call, or when encountering memory pressure.

What is going on?

Notes:

The mountpoint is definitely mounted with the async operation (which is the default).

To generate fsync requests, I used Postgresql's test_fsync tool. It uses a variety of ways to issue syncs and reports on the benchmarks so you can determine which is best for you on your system.

Timing differences with test_fsync indicate the fsync function takes longer to execute on async mounts than sync mounts, presumably because with sync mounts, data is being flushed all the time, and it's only when fsync is called that the data is flushed. However, the timing differences is quite erratic and I might just be encountering performance transience.

Mounting the server with the sync option changed nothing.

UPDATE: The plot thickens.

On Ubuntu/Mint 17, Linux kernel 3.13.0, (nfs version: 1.2.8), i set up a loopback mountpoint, with both sync and async options and re-ran the tests. Speed differences definitely show a difference between sync and async. nfsstat shows that after each run of pg_fsync_test, exactly 1 COMMIT transpired.

W.T.F.

Best Answer

A colleague at work dug up a possible answer:

the Netapp technical report tr-3183 (Using Red Hat Client with NetApp Storage over NFS) says:

Data ONTAP's NFS server promotes all write requests to FILE_SYNC, whether the client requests UNSTABLE, DATA_SYNC or FILE_SYNC writes. Thus the client never has to send a COMMIT request when it is writing to NetApp storage because all the writes are logged in the NVRAM and the clients get an acknowledgement for the writes right away. Hence the writes with NetApp storage are asynchronous in nature and are much faster.

Other things to try

I typically specify my hosts in the /etc/exports like this:

/cobbler/isos   192.168.1.0/24(rw,no_root_squash)

So for you with a single host IP:

/mnt/nfs01/volnfs01/share02 192.0.2.1/32(rw,anonuid=96,anongid=96,secure,root_squash,wdelay,sync)  192.0.2.2/32(rw,anonuid=96,anongid=96,secure,root_squash,wdelay,sync)
/mnt/nfs01/volnfs01/share01 192.0.2.1/32(rw,anonuid=96,anongid=96,secure,root_squash,wdelay,sync)  192.0.2.2/32(rw,anonuid=96,anongid=96,secure,root_squash,wdelay,sync)

nfs related servcies

Make sure that nfslock and other related services are both running on 192.0.2.2.
If you're using jumbo frames, be sure that ping -s <jumbo_mtu> 192.0.2.3 works from 192.0.2.2

NFS ACL Defaults – Not Obeyed on Copy

I think ACLs are sporadically supported in various ways within NFS. See this article on the NFS project's website.

http://www.linux-nfs.org/wiki/index.php/ACLs

If this isn't the issue, then I'd be extremely suspicious of cp. I seem to recall a Q&A regarding the command cp not fully copying the ACLs irregardless of the targeted mount type.

I believe it was this U&L Q&A titled: Using setfacl to allow group members to write to any file in a directory which led me to this SF Q&A titled: Why does cp not respect ACLs?.

Ironically our own @Gilles wrote an answer on that SF Q&A which explains why cp doesn't support the propagation of ACLs. I believe this is still the current situation!

excerpt from @Gilles answer

If cp creates the destination file, it replicates the permissions of the source file, except for the bits that are set in the umask. This is standard behavior (see e.g. step 3.b in the Single Unix v3 (POSIX 2001) specification.

Why was cp designed this way? Because there are many cases where this behavior is desirable, for example preserving a file's privacy when the original permissions are restrictive, and preserving executability is almost always the right thing to do. It is however unfortunate that not even GNU cp has an option to turn this behavior off.

Most copy tools (e.g. pax, rsync) behave in the same way. You can ensure the file will be created with the default permission by decoupling the source from the destination, for example with cat foo/baz.

Example

I setup the following file, afile and added an ACL to it.

$ touch afile
$ setfacl -m user:sam:rwx,group:users:rwx afile

We now have:

$ getfacl afile 
# file: afile
# owner: root
# group: root
user::rw-
user:sam:rwx
group::r--
group:users:rwx
mask::rwx
other::r--

When I copy these files to an NFSv3 share:

$ cp afile ~sam/
$ getfacl ~sam/afile 
getfacl: Removing leading '/' from absolute path names
# file: home/sam/afile
# owner: root
# group: root
user::rw-
group::rwx
other::r--

ACLs were lost. Trying to use the --preserve switches to cp:

$ cp --preserve afile ~sam/
cp: preserving permissions for `/home/sam/afile': Operation not supported
cp: preserving ACL for `/home/sam/afile': Operation not supported

Enabling ACL on NFS

Enabling ACL on the NFS mount seem to have no effect either:

mulder:/export/r1/home/sam on /home/sam type nfs (rw,intr,tcp,nfsvers=3,acl,rsize=16384,wsize=16384,addr=192.168.1.1)

$ cp --preserve afile ~sam/
cp: preserving permissions for `/home/sam/afile': Operation not supported
cp: preserving ACL for `/home/sam/afile': Operation not supported

Same HDD worked

Interestingly the --preserve switch did work when copying the file locally on the same EXT4 mounted drive.

$ cp --preserve afile afile2
$ getfacl afile2
# file: afile2
# owner: root
# group: root
user::rw-
user:sam:rwx
group::r--
group:users:rwx
mask::rwx
other::r--

Path forward?

In my research and experimentation it would appear that anything below NFSv4 does not support ACLs. The cp command seems able to preserve the ACLs as long as the underlying filesystem supports ACLs.

I found this article: Projects: NFS Version 4 Open Source Reference Implementation, which discusses the use of ACLs within NFSv4. So I would expect that copying ACLs to an NFSv4 share might be possible, I don't believe it's possible using NFSv2 or NFSv3 though.

References

Access Control Lists in Linux