Ubuntu – How to ensure nfs-server is not started until it can resolve hostnames? (16.10)

16.10networkingnfsserversystemd

Summary: Is there a way to make sure that the NFS server isn't started by systemd until it can properly resolve client machine names specified in /etc/exports?

Problem description:
I have found that NFS shares are not made properly available after the server (running 16.10) reboots. Clients get "access denied by server" errors, until exportfs -ra or service nfs-server restart is manually run on the server. After that, everything works as expected.

The server's /etc/exports contains only:

/mnt/raidarray clientmachine(rw)

where clientmachine is the hostname of the NFS client machine on the local network.

Problem identification: The output of systemctl status nfs-server (below) makes the problem clear: the name of the client can't be resolved at the time that the NFS server was started. 

● nfs-server.service - NFS server and services
Loaded: loaded (/lib/systemd/system/nfs-server.service; enabled
Active: active (exited) since Tue 2017-01-17 16:47:38 CST; 26min ago
Main PID: 1520 (code=exited, status=0/SUCCESS)
Tasks: 0 (limit: 4915)
CGroup: /system.slice/nfs-server.service
Jan 17 16:47:38 servermachine exportfs[1511]: exportfs: Failed to resolve clientmachine
Jan 17 16:47:38 servermachine systemd[1]: Started NFS server and services.

NetworkManager-wait-online.service is enabled, which I had understood serves to make sure that network.target (on which nfs-server depends) is not satisfied until network-online.target is satisfied.

Things that did not help:

  1. In case I misunderstood what NetworkManager-wait-online.service does, I tried adding an explicit After=network-online.target and Wants=network-online.target to nfs-server.service. This doesn't fix anything. I see the new dependency show up in systemctl list-dependencies, but name resolution still fails at boot. So it seems that network-online.target doesn't guarantee that hostname resolution can happen.

  2. Some googling suggested that requiring nss-lookup.target would ensure that network resolution is available, but adding it as a Wants and/or After dependency to nfs-server.service doesn't fix things either!

  3. Adding a Wants and/or After dependency on systemd-resolved.service instead of nss-lookup.target doesn't fix things either.

After adding all of these dependencies, nfs-server starts up very very late in the boot process (just before the desktop login things), but it still can't resolve hosts. Based on systemd-analyze plot, it appears that nmbd is wedged around this time, but I don't know if that's related.

Configuration information: This is on a desktop version of kubuntu 16.10 that is basically a fresh install.

NetworkManager.service is enabled and systemd-networkd.service is disabled — I didn't change that from the default.

Here is the NetworkManager-wait-online service definition:

[Unit]
Description=Network Manager Wait Online
Documentation=man:nm-online(1)
Requisite=NetworkManager.service
After=NetworkManager.service
Before=network-online.target
[Service]
Type=oneshot
ExecStart=/usr/bin/nm-online -s -q --timeout=30
RemainAfterExit=yes
[Install]
WantedBy=network-online.target

As a workaround I could hard-code IP addresses instead of hostnames into /etc/exports or /etc/hosts, but this seems somewhat brittle. Is there a better answer?

EDIT – Update: following @muru's advice below, I tried making a script to wait to resolve hostnames — just to see how long it takes. My script has to wait tens of seconds after systemd-resolved starts before it can actually resolve a host. This is very bizarre. I wonder if it's actually a local networking issue? Sounds like maybe a hard-coded (and auto-updating, perhaps, as suggested by @mark-stosberg) /etc/exports or /etc/hosts file is warranted.

Best Answer

In terms of what you are trying to do with systemd, you've already tried to do the best thing, which is to set your service to start after network-online. I read a lot of a systemd questions, and others have reported problems with the network not being fully online when they do this.

I recommend the idea of putting IP addresses in /etc/exports or /etc/hosts as you recommend. This simplifies an important disk mounting task during boot-up.

To make the system more robust, you could use a cron job or systemd timer that checks periodically to see if the DNS for the interesting hosts have changed. If so, automatically update /etc/hosts with the new values.

Related Question