Systemd – How to Debug Ordering Cycles in Systemd

Your question contains of two separate issues: (a) how to translate services dependency to systemd unit files syntax and (b) how to change unit definitions without changing unit files provided by software vendor.

As for (a), you correctly note that you want After= keyword. JBoss unit file should contain both After=<WebSphereMQ unit name> and Wants=<WebSphereMQ unit name>. Reason for that is: functional dependencies between units and order of units start are separate variables/dimensions. Sometimes you want service A to start every time when service B starts, and that's what Wants= and Requires= are for. Sometimes you want service A to start after service B, if service B is in queue already; that's what After= and Before= are for. Usually you want both of them.

As for (b), you need to use feature called "drop-ins". They are available in systemd since pretty much beginning, but documentation was scattered across many man pages and hard to find. This changed in one of more recent versions (219, I believe), when detailed description was added near end of man systemd.unit.

Drop-ins are small configuration bits that override parts of existing unit definition. Their goal is to allow system administrator to make local changes without having to overwrite unit file provided by software vendor (therefore software updates are a little easier, as you don't have to worry about package manager overwriting your changes).

Drop-ins are text files with names ending with .conf and are placed in <unit name>.d directory.

In your case, you want to create directory /etc/systemd/system/jboss.service.d/ (assuming JBoss comes with jboss.service unit file) and then create file webspheremq.conf with following content:

[Unit]
Wants=WebSphereMQ
After=WebSphereMQ

After reloading systemd configuration (systemctl daemon-reload) you should see changes in systemctl show jboss.service and systemctl status jboss.service. Here is what you can expect from second command (note "Drop-In" line):

● rsnapshot@monthly.service - rsnapshot monthly backup
   Loaded: loaded (/etc/systemd/system/rsnapshot@.service; static)
  Drop-In: /etc/systemd/system/rsnapshot@monthly.service.d
           └─after.conf
   Active: inactive (dead)

I marked your question as a duplicate.

Read JdeBP great answer here: How does systemd use /etc/init.d scripts?

Received wisdom is that the System V rc scripts must have an LSB header, and are run in parallel without honouring the priorities imposed by the /etc/rc?.d/ system. This is incorrect on all points.

In fact, they don't need to have an LSB header, and if they do not systemd-sysv-generator can recognize the more limited old RedHat comment headers (description:, pidfile:, and so forth). Moreover, in the absence of an LSB header it will fall back to the contents of the /etc/rc?.d symbolic link farms, reading the priorities encoded into the link names and constructing a before/after ordering from them, serializing the services. Not only are LSB headers not a requirement, and not only do they themselves encode before/after orderings that serialize things to an extent, the fallback behaviour in their complete absence is actually significantly non-parallelized operation.

The reason that /etc/rc3.d didn't appear to matter is that you probably had that script enabled via another /etc/rc?.d/ directory. systemd-sysv-generator translates being listed in any of /etc/rc2.d/, /etc/rc3.d/, and /etc/rc4.d/ into a native Wanted-By relationship to systemd's multi-user.target. Run levels are "obsolete" in the systemd world, and you can forget about them.

The question of what the search path is, is in fact actually addressed by what you quoted in your question:

systemd-sysv-generator generates the service units that run the System V rc scripts from /etc/init.d, if it doesn't find a native systemd service unit by that name already existing in the other six locations.

In more detail:

• For the scripts themselves:
  • systemd-sysv-generator looks for an environment variable named SYSTEMD_SYSVINIT_PATH.
    • If it exists, it is expected to be a conventional colon-separated search path of directories that systemd-sysv-generator scans one by one.
    • If it does not exist, or if it has an empty value, systemd-sysv-generator falls back upon a compiled-in default, which in vanilla systemd is the single directory /etc/init.d/.
  • Each directory is scanned for regular files that have the owner execute permission bit set on, with earlier directories in the path overriding later ones (a mechanism not really exercised in the default case of just the one directory in the search path) and the existence of service units with the same names inhibiting the generation of these nonce ones.
• For the "symbolic link farms":
  • There is a similar environment variable named SYSTEMD_SYSVRCND_PATH.
    • If it exists, it is expected to be a conventional colon-separated search path of directories that systemd-sysv-generator scans one by one.
    • If it does not exist, or if it has an empty value, systemd-sysv-generator falls back upon a compiled-in default, which in vanilla systemd is the single directory /etc/rc.d/.
  • Each directory is scanned for subdirectories named rc[2345].d, which are in turn scanned for directory entries that begin with S and two decimal digits and that are at least 4 characters long. The directory entries are not dereferenced.

Notice that systemd-sysv-generator …

• … does not look at the K* stuff;
• … does not check that the symbolic link farms are actually filled with symbolic links (They could be character special files or FIFOs for all that it cares, as it only looks at the names.); and
• … does not process the runlevel information from LSB comments (i.e. the Default-Start: and Default-Stop: comments).

Remember, in respect of the third point, that in vanilla van Smoorenburg rc the runlevel information from LSB comments does not directly control bootstrap/shutdown behaviour. That's the remit of the symbolic link farm directories. Describing van Smoorenburg rc in systemd terminology: the symbolic link farms are the service enable controls, that determine what services are automatically started at bootstrap; whereas the Default-Start: and Default-Stop: comments are (very approximately) service preset controls, that are turned into the enable controls.