updates: 5 (20171209)

updates: 5 (20171210)

• mount -t nfs4 [SERVER IP]:/archlinux /mnt works.
• ss -ntp | grep 2049 the client establishes a connection to the server before systemd begins.
• NSF4 id mapper can only be used with Kerberos?

the problem

I am attempting to set up a diskless node/workstation/system. The OS (4.13.12-1-ARCH) is installed on the SERVER /srv/archlinux. After a successful netboot from GRUB to NFSv4, systemd begins but fails at multiple stages, for example:

• Failed to mount Kernel Configuration File System.
• Failed to mount Kernel Debug File System.
• Failed to mount Huge Pages File System
• Failed to start Load/Save Random Seed.
• Failed to mount /tmp.
• Failed to start Rebuild Journal Catalog.
• Then ends with Not tainted 4.13.12-1-ARCH #1...

Or,

• Failed to mount POSIX Message Queue File System.
• Failed to start Remount Root and Kernel File System.
• Failed to mount Huge Pages File System.
• Failed to mount Kernel Debug File System.
• Failed to mount Kernel Configuration File System.
• Then ends with Not tainted 4.13.12-1-ARCH #1...

I suspect the failures are caused by an incorrect configuration of NFSv4 or the local network.

rpc.idmapd

/etc/idmapd.conf
  [General]
  Verbosity = 7
  Pipefs-Directory = /var/lib/nfs/rpc_pipefs
  Domain = localdomain
  [Mapping]
  Nobody-User = nobody
  Nobody-Group = nobody
  [Translation]
  Method = nnswitch

/etc/exports
(printed using # exportfs -v)
  /srv            <world>(rw,sync,wdelay,hide,no_subtree_check,fsid=0,sec=sys,no_root_squash,no_all_squash)
  /srv/archlinux  <world>(rw,sync,wdelay,hide,no_subtree_check,sec=sys,no_root_squash,no_all_squash)

(Exposed to "world" for debugging purposes)

Running rpc.idmapd -fvvv on a separate tty during bootup logs the following:

rpc.idmapd: libnfsidmap: using domain: localdomain
rpc.idmapd: libnfsidmap: Realms list: 'LOCALDOMAIN'
rpc.idmapd: libnfsidmap: processing 'Method' list
rpc.idmapd: libnfsidmap: loaded plugin /usr/lib/libnfsidmap/nsswitch.so for method nsswitch
rpc.idmapd: Expiration time is 600 seconds.
rpc.idmapd: Opened /proc/net/rpc/nfs4.nametoid/channel
rpc.idmapd: Opened /proc/net/rpc/nfs4.idtoname/channel
rpc.idmapd: nfsdcb: authbuf=* authtype=user
rpc.idmapd: nfs4_uid_to_name: calling nsswitch->uid_to_name
rpc.idmapd: nfs4_uid_to_name: nsswitch->uid_to_name returned 0
rpc.idmapd: nfs4_uid_to_name: final return value is 0
rpc.idmapd: Server : (user) id "0" -> name "root@localdomain"

If exportfs sec=sys, it continues like:

rpc.idmapd: nfsdch: authbuf=* authtype=user
rpc.idmapd: nfs4_name_to_uid: calling nsswitch->name_to_uid
rpc.idmapd: nss_getpwnam: name '0' domain 'localdomain': resulting localname '(null)'
rpc.idmapd: nss_getpwnam: name '0' does not map into domain 'localdomain'
rpc.idmapd: nfs4_name_to_uid: nsswitch->name_to_uid returned -22
rpc.idmapd: nfs4_name_to_uid: final return value is -22
rpc.idmapd: Server : (user) name "0" -> id "99"
(stops here)

+(20171209) After making sure that the /etc/hostname for the CLIENT was set to client2 (duh), if exportfs sec=none or sec=sys, it continues like:

rpc.idmapd: nfsdch: authbuf=* authtype=group
rpc.idmapd: nfs4_gid_to_name: calling nsswitch->gid_to_name
rpc.idmapd: nfs4_gid_to_name: nsswitch->gid_to_name returned 0
rpc.idmapd: nfs4_gid_to_name: final return value is 0
rpc.idmapd: Server : (group) id "190" -> name "systemd-journal@localdomain"
rpc.idmapd: nfsdch: authbuf=* authtype=user
rpc.idmapd: nfs4_name_to_uid: calling nsswitch->name_to_uid
rpc.idmapd: nss_getpwnam: name '0' domain 'localdomain': resulting localname '(null)'
rpc.idmapd: nss_getpwnam: name '0' does not map into domain 'localdomain'
rpc.idmapd: nfs4_name_to_uid: nsswitch->name_to_uid returned -22
rpc.idmapd: nfs4_name_to_uid: final return value is -22
rpc.idmapd: Server : (user) name "0" -> id "99"
(stops here)

If I instead change method from nsswitch to static (UID mapping in NFS)

/etc/idmapd.conf
  ...
  [Translation]
  Method = static
  [Static]
  root@localdomain = root

The rpc.idmapd -fvvv on a separate tty during bootup logs the following:

rpc.idmapd: libnfsidmap: using domain: localdomain
rpc.idmapd: libnfsidmap: Realms list: 'LOCALDOMAIN'
rpc.idmapd: libnfsidmap: processing 'Method' list
rpc.idmapd: static_getpwnam: name 'root@localdomain' mapped to 'root'
rpc.idmapd: static_getpwnam: group 'root@localdomain' mapped to ' root'
rpc.idmapd: libnfsidmap: loaded plugin /usr/lib/libnfsidmap/static.so for method static
rpc.idmapd: Expiration time is 600 seconds.
rpc.idmapd: Opened /proc/net/rpc/nfs4.nametoid/channel
rpc.idmapd: Opened /proc/net/rpc/nfs4.idtoname/channel
rpc.idmapd: nfsdcb: authbuf=* authtype=user
rpc.idmapd: nfs4_uid_to_name: calling static->uid_to_name
rpc.idmapd: nfs4_uid_to_name: static->uid_to_name returned 0
rpc.idmapd: nfs4_uid_to_name: final return value is 0
rpc.idmapd: Server : (user) id "0" -> name "root@localdomain"

If exportfs sec=sys, it continues like:

rpc.idmapd: nfsdch: authbuf=* authtype=user
rpc.idmapd: nfs4_name_to_uid: calling static->name_to_uid
rpc.idmapd: nfs4_name_to_uid: static->name_to_uid returned -2
rpc.idmapd: nfs4_name_to_uid: final return value is -2
rpc.idmapd: Server : (user) name "0" -> id "99"
(stops here)

If exportfs sec=none, it continues like:

rpc.idmapd: nfsdch: authbuf=* authtype=group
rpc.idmapd: nfs4_gid_to_name: calling static->gid_to_name
rpc.idmapd: nfs4_gid_to_name: static->gid_to_name returned -2
rpc.idmapd: nfs4_gid_to_name: final return value is -2
rpc.idmapd: Server : (group) id "190" -> name "nobody"
rpc.idmapd: nfsdch: authbuf=* authtype=user
rpc.idmapd: nfs4_name_to_uid: calling static->name_to_uid
rpc.idmapd: nfs4_name_to_uid: static->name_to_uid returned -2
rpc.idmapd: nfs4_name_to_uid: final return value is -2
rpc.idmapd: Server : (user) name "0" -> id "99"
(stops here)

Similar problems with the user ID mapping:

• NFSv4 User Mapping
• NFS user mapping
• Mapping UID and GID of local user to the mounted NFS share
• And many many more… Often related to a switch from NFSv3 to NFSv4, and rarely about netboot.

troubleshooting

• No firewall
• No Kerberos, LDAP, etc.
• No SELinux
• The user root exists on both SERVER and CLIENT, with the same password.

SERVER

All other relevant configuration files for NFSv4 I could identify on the SERVER.

/etc/nsswitch.conf
  passwd: compat mymachines systemd
  group: compat mymachines systemd
  shadow: compat
  publickey: files
  hosts: files mymachines resolve [!UNAVAIL=return] dns myhostname
  networks: files
  protocols: files
  services: files
  ethers: files
  rpc: files
  netgroup: files

/etc/nfs.conf
  (all settings commented out)
/etc/conf.d/nfs-common.conf
  (all settings commented out)

network configuration

• How to set the domain name on GNU/Linux?
• Archlinux Wiki Network configuration: Set the hostname
• Archlinux Wiki Network configuration: Local network hostname resolution

The SERVER hostname is server and has 3 network devices (nd[1-3]). The Gateway default via 192.168.0.1 nd1.

/etc/hosts
  127.0.0.1      localhost.localdomain  localhost
  ::1            ip6.localhost          localhost
  192.168.0.101  nd1.localdomain        server servernd1
  192.168.1.101  nd2.localdomain        server servernd2
  192.168.2.101  nd3.localdomain        server servernd2
  192.168.1.102  client1.localdomain    client1
  192.168.2.102  client2.localdomain    client2

/etc/resolveconf.conf
  name_servers=192.168.0.1

# hostname -f
# nd1.localdomain

# hostname -i
192.168.0.101 192.168.1.101 192.168.2.101

# getent hosts IP -> the corresponding line in /etc/hosts
# getent ahosts HOSTNAME -> the corresponding line in /etc/hosts

# ping -c 3 server.localdomain -> 0% packet loss

# id -u root -> 0
# id -un 0 -> root

Display the system's effective NFSv4 domain name on stdout.
# nfsidmap -d -> localdomain

Display on stdout all keys currently in the keyring used to cache ID mapping results. These keys are visible only to the superuser.
# nfsidmap -l -> nfsidmap: '.id_resolver' keyring was not found.

CLIENT

/etc/hostname +(20171209)
  client2
/etc/hosts
  (exactly the same as the hosts file on the server)
/etc/resolveconf.conf
  name_servers=192.168.0.1
/etc/idmapd.conf
  (exactly the same as the idmapd.conf file on the server)
/etc/fstab
  # sys=sec or sys=none to correspond to server export settings. 
  /dev/nfs  /  nfs  rw,hard,rsize=9151,sec=sys,clientaddr=192.168.2.102  0  0
  devtmpfs  /dev   devtmpfs  defaults
  proc      /proc  proc      defaults
  none      /run   tmpfs     defaults
  sys       /sys   sysfs     defaults
  run       /run   tmpfs     defaults
  tmp       /tmp   tmpfs     defaults

The fstab was defined by comparing the mounted directories on the server using findmnt -A.

net_nfs4

• +(20171210) NFS version on SERVER and CLIENT cat /proc/fs/nfsd/versions -> -2 +3 +4 +4.1 +4.2
• On SERVER and CLIENT cat /sys/module/nfsd/parameters/nfs4_disable_idmapping -> N.
• On SERVER echo "options nfsd nfs4_disable_idmapping=0" > /etc/modprobe.d/nfsd.conf.
• On CLIENT the /sys/module/nfs/parameters/nfs4_disable_idmapping does not exist, and not sure how to manually create it as the /sys is read only.
• +(20171210) On CLIENT echo "options nfs nfs4_disable_idmapping=0" > /etc/modprobe.d/nfs.conf.

The CLIENT IP is 192.168.2.102/24. The CLIENT network device is connected to SERVER nd2 192.168.2.101/24 (hostname: servernd2).

The network information during boot:

:: running early hook [udev]
starting version 235
:: running hook [udev]
:: Triggering uevents...
:: running hook [net_nfs4]
IP-Config: eth0 hardware address [CLIENT NETWORK DEVICE MAC] mtu 1500 DHCP
hostname client2 IP-Config: eth0 guessed broadcast address 192.168.2.255
IP-Config: eth0 complete (from 192.168.0.101):
 address: 192.168.2.102     broadcast: 192.168.2.255     netmask: 255.255.255.0
 gateway: 192.168.2.101     dns0     : 192.168.0.1       dns1   : 0.0.0.0
 host   : client2
 domain : localdomain
 rootserver: 192.168.0.101 rootpath: /srv/archlinux
 filename  : /netboot/grub/i386-pc/core.0
NFS-Mount: 192.168.2.101:/archlinux
Waiting 10 seconds for device /dev/nfs ...
(systemd takes over from here)

Why the NSFv4 errors occur?

Server : (group) id "190" -> name "nobody"

With NFSv4, things change: users are mapped by username, and the mapping between user names and user IDs is handled by a process called "ID map daemon" (idmapd). In particular, NFSv4 clients and server should use the same domain for the mapping to work properly, otherwise requests will be mapped to the anonymous user/group.
— Trying out NFSv4 (on Linux and Solaris) — March 15th, 2012 – 13:03 / bronto

---

In an ideal world, the user and group of the requesting client would determine the permissions of the data returned. We don't live in an ideal world. Two real-world problems intervene:

1. You might not trust the root user of a client with root access to
   the server's files.
2. The same username on client and server might have different
   numerical ID's

Problem 1 is conceptually simple. John Q. Programmer is given a test machine for which he has root access. In no way does that mean that John Q. Programmer should be able to alter root owned files on the server. Therefore NFS offers root squashing, a feature that maps uid 0 (root) to the anonymous (nfsnobody) uid, which defaults to -2 (65534 on 16 bit numbers).
— NFS: Overview and Gotchas — Copyright (C) 2003 by Steve Litt

+(20171209) rpc.idmapd: nss_getpwnam: name '0' domain 'localdomain': resulting localname '(null)'

According to Steve Dickson in a comment (2011-08-12 16:01:55 EDT) to a Red Hat Bugzilla – Bug 715430 report

The [error] statement explains the problem. DNS on the local machine was
not set up (or returning NULL) and the Domain= variable in
/etc/idmapd.conf was not set.

nss_getpwnam: name '0' does not map into domain

On the Debian Mailing Lists, in an e-mail correspondence between Jonas Meurer and Christian Seiler (20150722) concerning "Kerberos-secured NFSv4" the error is explained in detail. My summary of the discussion:

When the NFS client sends nss_getpwnam: name '8' domain 'freesources.org': resulting localname '(null)'

The NFS client sends just the uid converted to a string in some cases instead of the properly translated NFS username, which the server then rejects.

The client should send nss_getpwnam: name 'mail@freesources.org' domain 'freesources.org': resulting localname 'mail'

Here you can see that the owner name that was transmitted by the NFS
client was 'mail@freesources.org' (and not simply '8'), so that does
contain an @; nss_getpwname can see that the domain name matches
and just strips it, resulting in a user name 'mail', which it looks
up in /etc/passwd, returns the user id (in this case, 8, because it's
the same on client and server) and the server is perfectly happy.

So why does the client send the wrong username?
… every once in a while, idmapping will fail, so the kernel will just send a number. But that number will cause the chown command to fail, since the server won't translate it
back.

Short answer: I have no idea.

Longer answer: …

If I understand the longer answer correctly, the problem could occur because the NFS client relies on the "kernel's key cache". For the NFS server this should never be a problem because the "kernel's key cache" is never used.

Nonetheless,

Since you are using just regular nsswitch via /etc/passwd, nss_getpwnam should never fail in your case, unless you do some weird stuff with /etc/passwd at the same time.

The answer also refers to an alternative method to idmapd; nfsidmap, although reading the man I cannot quite understand how it would replace idmapd.

+(20171209) nss_getpwnam: name 'root@domain.com' does not map into domain 'localdomain'

This error message does not seem to occur for me, I am however including the answer from SUSE's support knowledgebase — 10-DEC-13 Modified Date: 12-OCT-17 — because of the description of cause, and the proposed remedy which stands in contrast to the other found discussions.

NFSv4 handles user identities differently than NFSv3. In v3, an nfs client would simply pass a UID number in chown (and other requests) and the nfs server would accept that (even if the nfs server did not know of an account with that UID number). However, v4 was designed to pass identities in the form of @. To function correctly, that normally requires idmapd (id mapping daemon) to be active at client and server, and for each to consider themselves part of the same id mapping domain.

Chown failures or idmapd errors like the ones documented above are typically a result of either:

1. The username is known to the client but not known to the server, or
2. The idmapd domain name is set differently on the client than it is on the server.

Therefore, this issue can be fixed by insuring that the nfs server and client are configured with the same idmapd domain name (/etc/idmapd.conf) and both have knowledge of the usernames / accounts in question.

However, it is often not convenient to insure that both sides have the same user account knowledge, especially if the nfs server is a filer. The NFS community has recognized that this idmapd feature of NFSv4 is often more troublesome that it is worth, so there are steps and modifications coming into effect to allow the NFSv3 behavior to work even under NFSv4.

The proposed remedy is to disable idmapd.

nfs.nfs4_disable_idmapping=1

+(20171209) Wireshark

Analyzing the Wireshark log, it is quite extensive but begins with something like:

[IP CLIENT] -> [IP SERVER] NFS 226 V4 Call ACCESS FH: [HEX VALUE], [Check: RD LU MD XT DL]
[IP SERVER] -> [IP CLIENT] NFS 238 V4 Reply (Call In 34) ACCESS, [Allowed: RD LU MD XT DL]
[IP CLIENT] -> [IP SERVER] NFS 246 V4 Call LOOKUP DH: [HEX VALUE]/archlinux

where a similar pattern [A HEX VALUE]/[PATH] can be discerned for
/sbin, /usr, /bin, /init, /lib, /systemd, /dev, /proc, /sys, /run, /, /lib64.

When the CLIENT requests /Id-linux-x86-64.so.2 the first errors start to appear:

[IP CLIENT] -> [IP SERVER] NFS 342 V4 Call OPEN DH: [HEX VALUE]/Id-linux-x86-64.so.2
[SERVER IP] -> [CLIENT IP] NFS 166 V4 Reply (Call In 124) OPEN Status: NFS4ERR_SYMLINK

The pattern more or less repeats itself with more frequent errors, for example, LOOKUP Status; and OPEN Status: reporting NFS4ERR_NOENT.

Interestingly, it is at the very end of the log where to first and only reference to user permission is made,

[SERVER IP] -> [CLIENT IP] NFS 182 V4 Reply (Call In 9562) SETATTR Status: NFS4ERR_BADOWNER

RFC

According to

• RFC7530 (Network File System (NFS) Version 4 Protocol, 201503, PROPOSED STANDARD) — Updated by RFC7931
• RFC5661 (Network File System (NFS) Version 4 Minor Version 1 Protocol, 201001, PROPOSED STANDARD) — Updated by RFC8178
• RFC7862 (Network File System (NFS) Version 4 Minor Version 2 Protocol, 201001, PROPOSED STANDARD) — Updated by RFC8178 — which refers back to [RFC5661].

NFS4ERR_BADOWNER (Error Code 10039)

This error is returned when an owner or owner_group attribute value or the who field of an ACE within an ACL attribute value cannot be translated to a local representation.

The specifications discuss in Section 5.9. Interpreting owner and owner_group, I am not sure what to cite as relevant however.

NFS4ERR_SYMLINK (Error Code 10029)

The current filehandle designates a symbolic link when the current operation does not allow a symbolic link as the target.

NFS4ERR_NOENT (Error Code 2)

This indicates no such file or directory. The file system object referenced by the name specified does not exist.

The error could however be expected …

The current filehandle is assumed to refer to a regular directory a named attribute directory. LOOKUPP assigns the filehandle for its parent directory to be the current filehandle. If there is no parent directory, an NFS4ERR_NOENT error must be returned. Therefore, NFS4ERR_NOENT will be returned by the server when the current filehandle is at the root or top of the server's file tree.

+(20171210) mount -t nfs4 [SERVER IP]:/archlinux /mnt

On the client computer, using the Archlinux "LiveUSB" I was able to mount the network drive, download the latest kernel (4.14-4-1-ARCH) via the SERVER internet connection, and install archlinux on the [SERVER IP]/archlinux.

During install rpc.idmapd -fvvv indicated a successful mapping of usernames, for example,

rpc.idmapd: Server : (user) id "0" -> name "root@localdomain"
rpc.idmapd: Server : (group) id "99" -> name "nobody@localdomain"
... -> name "tty@localdomain"
... -> name "systemd-journal-upload@localdomain"
... -> name rpc@localdomain
... -> name systemd-journal@localdomain
... -> name utmp@localdomain

The result of genfstab was also different:

[SERVER IP]:/archlinux  /  nfs4 rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,times=600,retrans=2,sec=sys,clientaddr=[CLIENT IP],local_lock=none,addr=[SERVER IP]  0 0

Nevertheless, after reboot systemd failed again with the same failures as described at the beginning of the post.

+(20171210) Is the remote directory on the server mounted to /new_root?

The mkinitcpio script uses the variable mount_handler to carry an assigned "mounting function", in this case nfs_mount_handler(), to which the "root path" is passed $1 at a later stage; /new_root.

I am trying to verify that the client has mounted the [SERVER IP]:/archlinux to the /new_root. On the server, I can only observe that the client has established a connection but not if the directory is mounted and to where?

showmount -a server -> All mount points on server: (empty)

ss -ntp | grep 2049 ->
ESTAB  0    0   192.168.2.101:2049  192.168.2.102:809 (random port)

+(20171210) NFS4, sec=sys and id mapper are incompatible?

Reading the doco, it looks like sec=sys and the id mapper can be used to correctly map uid/gid to name where the client and server have different mappings in /etc/passwd and /etc/group. This simply isn't true.

That's because with sec=sys the id mapper doesn't come into play in the authentication part of the nfs protocol, only the file attributes part. With sec=sys authentication, nfs just passes the client uid/gid which is used directly by the server. So permissions checks will be screwed if client and server uid and gid don't align. To confuse things further, when the client creates a new file it is the authentication credentials that are used, so the file gets created at the server with the client's uid/gid. After that nfs uses idmap to get the file attributes, so the uid/gid (which originally came from the client) gets mapped at the server, and you end up seeing the server's name for a client uid/gid. Borkage! On the other hand, if the file was originally created at the server, you will see the correct name at the client, even if the uid/gid differs. But permissions checking will still be broken. — kimmie — Posted: Wed Feb 20, 2013 3:14 am Post subject: — Emphasis in original

echo "N" > /sys/module/nfsd/parameters/nfs4_disable_idmapping

echo "N" > /sys/module/nfs/parameters/nfs4_disable_idmapping

options nfsd nfs4_disable_idmapping=N

options nfs nfs4_disable_idmapping=N