Linux Permissions – Fix ‘Permission Denied’ When Using `rm` on NFS Mount

linuxnfspermissions

Context:

  • Storage is a Kubernetes Persistent volume, provided by a TrueNAS NAS and accessed via NFS (with k8s accessMode ReadWriteMany)
  • I have checked the Permissions for the TrueNAS Dataset and confirmed that RWX are set for User, Group, and Other (ack that this is not the most secure, but it's sensitive data on a home network – I would like to "do it right", but that's not a priority)
  • Debug pod (mentioned below) is using ubuntu image
  • There is no /var/log/audit/audit.log file on the debug pod, so I assume that SELinux is not involved (though, from the phrasing of that statement, you can probably correctly infer that I'm no expert at SELinux)

I found a file on a mounted NFS drive that was unnecessary and should be deleted. I started up a new pod and mounted the PVC to it, but when I tried to rm the file – even when I used sudo – I got permission denied. I also noted the unusual ownership of the file (nobody) and tried changing that in case that was the cause, which was also forbidden:

$ ls -al
...
-rwxr-xr-x   1 nobody root    1016756 Nov 26 01:21  DELETEME.txt
$ rm DELETEME.txt
rm: cannot remove 'DELETEME.txt': Permission denied
$ sudo rm DELETEME.txt
rm: cannot remove 'DELETEME.txt': Permission denied
$ sudo chown root:root DELETEME.txt
chown: changing ownership of 'DELETEME.txt': Operation not permitted

In case this was due to some aspect of Kubernetes, I also tried directly mounting the NFS share (with -o rw) to a (non-k8s-pod) machine and carrying out the same commands, with the same results.

What could be preventing root from deleting or modifying this file?

EDIT 1 – AFAICT, the NFS share is exported with 777 permissions from TrueNas:

screenshot of TrueNAS UI showing 777 permissions

and the contents of /etc/exports on the TrueNAS box:

# cat /etc/exports
V4: / -sec=sys
/mnt/low-resiliency-with-read-cache/ombi-data -mapall="k8s-user":"k8s-user"
/mnt/high-resiliency/k8s/nfs/vols/pvc-cd6d0f5d-a7d0-47a9-8371-29ab9ca27764 -maproot="root":"wheel"
/mnt/high-resiliency/manual-nfs
/mnt/high-resiliency/k8s/nfs/vols/pvc-a485ee75-9413-4b26-abdd-e5c3c23ef7ba -maproot="root":"wheel"

Best Answer

Dunno what TrueNAS’ web interface does, but I can tell you what your /etc/exports says, if the export in question truly is /mnt/low-resiliency-with-read-cache/ombi-data (which I kind of doubt, given the other names):

-mapall="k8s-user":"k8s-user"`

This means that all remote users, root, not root, whatever, are mapped to k8s-user. So you can forget about sudo, it won’t do anything.

Deleting a file isn’t about modifying the file though, it is about modifying the directory. Unfortunately, your question doesn’t have all information to give a 100% answer. But I bet k8s-user does not have write access to the directory containing this file either.

And then there’s also the possibility that Access Control Lists are in use. They go beyond what classic Linux/UNIX file modes do. They could even affect those other exports, which have -maproot=root:wheel.

Related Question