Centos – mkdir gives different errors (permission denied vs. file exists) depending on whether directory has been recently accessed

centosfilesystemsmountnfspermissions

problem

First, I log in to a new workstation.

myAcct@ws5: mkdir /users/adminAcct/foo/parentDir/childDir
mkdir: cannot create directory ‘/users/adminAcct/foo/parentDir/childDir’: Permission denied

A Permission denied error, even though the childDir exists.

If I access childDir, this error changes. I can do this in iPython with os.path.isdir() or in the shell with ls.

myAcct@ws5: ls /users/adminAcct/foo/parentDir/childDir
file.cfg
myAcct@ws5: mkdir /users/adminAcct/foo/parentDir/childDir
mkdir: cannot create directory ‘/users/adminAcct/foo/parentDir/childDir’: File exists

I want to be able to consistently return a 'File exists' error.

background

I have --x group permission on the parent directory, /users/adminAcct/foo/parentDir. The child directory /users/adminAcct/foo/parentDir/childDir exists, and I have r-x group permissions on that directory. This originally started as a problem in python, but I was able to recreate it in the shell. I'm working on CentOS 6 workstations that are part of a cluster that has home directories mounted via NFS. The /users directory is in /etc/fstab, and is mounted with the options (rw,vers=3,hard,intr,addr=<some IP addr>). Kernel is Linux ws5.MyCompany.com 2.6.32-431.17.1.el6.x86_64 #1 SMP Wed May 7 23:32:49 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

context

For some time period after I access the child directory, mkdir will continue to return File exists errors. After some period, mkdir will once again return Permission denied errors. This creates intermittent problems starting and running airflow CeleryExecutor workers on these workstations, using /users/adminAcct/foo/parentDir/childDir as AIRFLOW_HOME. I haven't been able to predict when these problems will re-appear.

Best Answer

The NFS client caches attributes, which have a wall clock based timeout.

This improves performance but gives 'slightly incorrect' results in some scenarios. The standard UNIX garantuees about file existance etc. are not always met by nature of the way that the client works.

One way to alleviate this is to disable client side caching (noac), or just directory entry caching (lookupcache=none) -- see https://linux.die.net/man/5/nfs

Related Question