How to a systemd service flag that is is ready, so that other services can wait for it to be ready before they start

systemd

I have a bunch of services (say C0, C1, … C9) that should only start after a service S has completed its initialization and is fully running and ready for the other services. How do I arrange that with systemd?

In Ordering services with path activation and target in systemd it is assumed that service S has a mechanism for writing out some sort of flag file. Assume here, in contrast, that I have full control over the program that service S runs, and can add systemd mechanisms into it if needs be.

Best Answer

One doesn't necessarily need this.

If the C services need to wait for S to be ready so that they can open a socket connection to it, then one doesn't necessarily need to do this at all. Rather, one can take advantage of early listening socket opening by service managers.

Several systems, including Laurent Bercot's s6, my nosh toolset, and systemd, have ways in which a listening socket can be opened early on, the very first thing in setting up the service. They all involve something other than the service program opening the listening socket(s), and the service program, when invoked, receiving the listening sockets(s) as already-open file descriptors.

With systemd, specifically, one creates a socket unit that defines the listening socket. systemd opens the socket unit and sets it up so that the kernel networking subsystem is listening for connections; and passes it to the actual service as an open file descriptor when it comes to spawn the process(es) that handle(s) connections to the socket. (It can do this in two ways, just like inetd could, but a discussion of the details of Accept=true versus Accept=false services is beyond the scope of this answer.)

The important point is that one does not necessarily need more ordering than that. The kernel batches up client connections in a queue until the service program is initialized, and ready to accept them and talk to clients.

When one does, readiness protocols are the thing.

systemd has a set of readiness protocols that it understands, specified service by service with the Type= setting in the service unit. The particular readiness protocol of interest here is the notify readiness protocol. With it, systemd is told to expect messages from the service, and when the service is ready it sends a message that flags readiness. systemd delays the activation of the other services until readiness is flagged.

Making use of this involves two things:

  • Modifying the code of S so that it calls something like Pierre-Yves Ritschard's notify_systemd() function or Cameron T Norman's notify_socket() function.
  • Setting up the service unit for the service with Type=notify and NotifyAccess=main.

The NotifyAccess=main restriction (which is the default) is because systemd needs to know to ignore messages from mischievous (or just plain faulty) programs, because any process on the system can send messages to systemd's notification socket.

One uses Pierre-Yves Ritschard's or Cameron T Norman's code for preference because it does not exclude the possibility of having this mechanism on UbuntuBSD, Debian FreeBSD, actual FreeBSD, TrueOS, OpenBSD, and so forth; which the code supplied by the systemd authors does exclude.

One trap to avoid is the systemd-notify program. It has several major problems, not the least of which is that messages sent with it can end up being thrown away unprocessed by systemd. The most major problem in this case is that it doesn't run as the "main" process of the service, so one has to open up the readiness notifications for the service S to every process on the system with NotifyAccess=all.

Another trap to avoid is thinking that the forking protocol is simpler. It is not. Doing it correctly involves not forking and exiting the parent until (for one thing) all of the program's worker threads are running. This does not match how the overwhelming majority of dæmons that fork actually fork.

Further reading

Related Question