Init Systems – Pros and Cons of Upstart and systemd

initsystemdupstart

It appears systemd is the hot new init system on the block, same as Upstart was a few years ago. What are the pros/cons for each? Also, how does each compare to other init systems?

Best Answer

2016 Update

Most answers here are five years old so it's time for some updates.

Ubuntu used to use upstart by default but they abandoned it last year in favor of systemd - see:

Because of that there is a nice article Systemd for Upstart Users on Ubuntu wiki - very detailed comparison between upstart and systemd and a transition guide from upstart to systemd.

(Note that according to the Ubuntu wiki you can still run upstart on current versions of Ubuntu by default by installing the upstart-sysv and running sudo update-initramfs -u but considering the scope of the systemd project I don't know how it works in practice, or whether or not systemd is possible to uninstall.)

Most of the info in the Commands and Scripts sections below is adapted from some of the examples used in that article (that is conveniently licensed just like Stack Exchange user contributions under the Creative Commons Attribution-ShareAlike 3.0 License).

Here is a quick comparison of common commands and simple scripts, see sections below for detailed explanation. This answer is comparing the old behavior of Upstart-based systems with the new behavior of systemd-based systems, as asked in the question, but note that the commands tagged as "Upstart" are not necessarily Upstart-specific - they are often commands that are common to every non-systemd Linux and Unix system.

Commands

Running su:

  • upstart: su
  • systemd: machinectl shell

(see "su command replacement" section below)

Running screen:

  • upstart: screen
  • systemd: systemd-run --user --scope screen

(see "Unexpected killing of background processes" section below)

Running tmux:

  • upstart: tmux
  • systemd: systemd-run --user --scope tmux

(see "Unexpected killing of background processes" section below)

Starting job foo:

  • upstart: start foo
  • systemd: systemctl start foo

Stopping job foo:

  • upstart: stop foo
  • systemd: systemctl stop foo

Restarting job foo:

  • upstart: restart foo
  • systemd: systemctl restart foo

Listing jobs:

  • upstart: initctl list
  • systemd: systemctl status

Checking configuration of job foo:

  • upstart: init-checkconf /etc/init/foo.conf
  • systemd: systemd-analyze verify /lib/systemd/system/foo.service

Listing job's environement variables:

  • upstart: initctl list-env
  • systemd: systemctl show-environment

Setting job's environment variable:

  • upstart: initctl set-env foo=bar
  • systemd: systemctl set-environment foo=bar

Removing job's environment variable:

  • upstart: initctl unset-env foo
  • systemd: systemctl unset-environment foo

Logs

In upstart, the logs are normal text files in the /var/log/upstart directory, so you can process them as usual:

cat /var/log/upstart/foo.log
tail -f /var/log/upstart/foo.log

In systemd logs are stored in an internal binary format (not as text files) so you need to use journalctl command to access them:

sudo journalctl -u foo
sudo journalctl -u foo -f

Scripts

Example upstart script written in /etc/init/foo.conf:

description "Job that runs the foo daemon"
start on runlevel [2345]
stop on runlevel [016]
env statedir=/var/cache/foo
pre-start exec mkdir -p $statedir
exec /usr/bin/foo-daemon --arg1 "hello world" --statedir $statedir

Example systemd script written in /lib/systemd/system/foo.service:

[Unit]
Description=Job that runs the foo daemon
Documentation=man:foo(1)
[Service]
Type=forking
Environment=statedir=/var/cache/foo
ExecStartPre=/usr/bin/mkdir -p ${statedir}
ExecStart=/usr/bin/foo-daemon --arg1 "hello world" --statedir ${statedir}
[Install]
WantedBy=multi-user.target

su command replacement

A su command replacement was merged into systemd in pull request #1022:

because, according to Lennart Poettering, "su is really a broken concept".

He explains that "you can use su and sudo as before, but don't expect that it will work in full".

The official way to achieve a su-like behavior is now:

machinectl shell

It has been further explained by Lennart Poettering in the discussion to issue #825:

"Well, there have been long discussions about this, but the problem is that what su is supposed to do is very unclear. [...] Long story short: su is really a broken concept. It will given you kind of a shell, and it’s fine to use it for that, but it’s not a full login, and shouldn’t be mistaken for one." - Lennart Poettering

See also:

Unexpected killing of background processes

Commands like:

no longer work as expected. For example, nohup is a POSIX command to make sure that the process keeps running after you log out from your session. It no longer works on systemd. Also programs like screen and tmux need to be invoked in a special way or otherwise the processes that you run with them will get killed (while not getting those processes killed is usually the main reason of running screen or tmux in the first place).

This is not a mistake, it is a deliberate decision, so it is not likely to get fixed in the future. This is what Lennart Poettering has said about this issue:

In my view it was actually quite strange of UNIX that it by default let arbitrary user code stay around unrestricted after logout. It has been discussed for ages now among many OS people, that this should possible but certainly not be the default, but nobody dared so far to flip the switch to turn it from a default to an option. Not cleaning up user sessions after logout is not only ugly and somewhat hackish but also a security problem. systemd 230 now finally flipped the switch and finally by default cleans everything up correctly when the user logs out.

For more info see:

High-level startup concept

In a way systemd works backwards - in upstart jobs start as soon as they can and in systemd jobs start when they have to. At the end of the day the same jobs can be started by both systems and in pretty much the same order, but you think about it looking from an opposite direction so to speak.

Here is how Systemd for Upstart Users explains it:

Upstart's model for starting processes (jobs) is "greedy event-based", i. e. all available jobs whose startup events happen are started as early as possible. During boot, upstart synthesizes some initial events like startup or rcS as the "tree root", the early services start on those, and later services start when the former are running. A new job merely needs to install its configuration file into /etc/init/ to become active.

systemd's model for starting processes (units) is "lazy dependency-based", i. e. a unit will only start if and when some other starting unit depends on it. During boot, systemd starts a "root unit" (default.target, can be overridden in grub), which then transitively expands and starts its dependencies. A new unit needs to add itself as a dependency of a unit of the boot sequence (commonly multi-user.target) in order to become active.

Usage in distributions

Now some recent data according to Wikipedia:

Distributions using upstart by default:

Distributions using systemd by default:

(See Wikipedia for up to date info)

Distributions using neither Upstart nor systemd:

Controversy

In the past A fork of Debian has been proposed to avoid systemd. The Devuan GNU+Linux was created - a fork of Debian without systemd (thanks to fpmurphy1 for pointing it out in the comments).

For more info about this controversy, see:

As many of you might know already, the Init GR Debian vote promoted by Ian Jackson wasn't useful to protect Debian's legacy and its users from the systemd avalanche.

This situation prospects a lock in systemd dependencies which is de-facto threatening freedom of development and has serious consequences for Debian, its upstream and its downstream.

The CTTE managed to swap a dependency and gain us time over a subtle install of systemd over sysvinit, but even this process was exhausting and full of drama. Ultimately, a week ago, Ian Jackson resigned. [...]

I am resigning from the Technical Committee with immediate effect.

While it is important that the views of the 30-40% of the project who agree with me should continue to be represented on the TC, I myself am clearly too controversial a figure at this point to do so. I should step aside to try to reduce the extent to which conversations about the project's governance are personalised. [...]

Devuan was born out of a controversy over the decision to use as the default init system for Debian. The official Debian position on systemd is full of claims that others have debunked. Interested readers can continue discussing this hot topic in The systemd controversy. However we encourage you to keep your head cool and your voice civil. At Devuan we’re more interested in programming them wrong than looking back. [...]

Some websites and articles dedicated to the systemd controversy has been created:

There is a lot of interesting discussion on Hacker News:

Similar tendencies in other distros can be observed as well:

Philosophy

upstart follows the Unix philosophy of DOTADIW - "Do One Thing and Do It Well." It is a replacement for the traditional init daemon. It doesn't do anything other than starting and stopping services. Other tasks are delegated to other specialized subsystems.

systemd does much more than that. In addition to starting and stopping services it also manages passwords, logins, terminals, power management, factory resets, log processing, file system mount points, networking and much more - see the NEWS file for some of the features.

Plans of expansion

According to A Perspective for systemd What Has Been Achieved, and What Lies Ahead presentation by Lennart Poettering in 2014 at GNOME.asia, here are the main objectives of systemd, areas that were already covered and those that were still in progress:

systemd objectives:

Our objectives

  • Turning Linux from a bag of bits into a competitive General Purpose Operating System.

  • Building the Internet’s Next Generation OS Unifying pointless differences between distributions

  • Bringing innovation back to the core OS

  • Desktop, Server, Container, Embedded, Mobile, Cloud, Cluster, . . . These areas are closer together than you might think

  • Reducing administrator complexity, reliability without supervision

  • Everything introspectable

  • Auto discovery, plug and play is key

  • We fix things where they are broken, never tape over them

Areas already covered:

What we already cover:

init system, journal logging, login management, device management, temporary and volatile file management, binary format registration, backlight save/restore, rfkill save/restore, bootchart, readahead, encrypted storage setup, EFI/GPT partition discovery, virtual machine/container registration, minimal container management, hostname management, locale management, time management, random seed management, sysctl variable management, console managment, . . .

Work in progress:

What we are working on:

  • network management
  • systemd-networkd
  • Local DNS cache, mDNS responder, LLMNR responder, DNSSEC verification
  • IPC in the kernel
  • kdbus, sd-bus
  • Time synchronisation with NTP
  • systemd-timesyncd
  • More integration with containers
  • Sandboxing of Services
  • Sandboxing of Apps
  • OS Image format
  • Container image format
  • App image format
  • GPT with auto-discovery
  • Stateless systems, instantiatable systems, factory reset
  • /usr is the OS
  • /etc is (optional) configuration
  • /var is (optional) state
  • Atomic node initialisation and updates
  • Integration with the cloud
  • Service management across nodes
  • Verifiable OS images
  • All the way to the firmware
  • Boot Loading

Scope of this answer

As fpmurphy1 noted in the comments, "It should be pointed out that systemd has expanded its scope of work over the years far beyond simply that of system startup."

I tried to include most of the relevant info here. Here I am comparing the common features of Upstart and systemd when used as init systems as asked in the question and I only mention features of systemd that go beyond the scope of an init system because those cannot be compared to Startup, but their presence is important to understand the difference between those two projects. The relevant documentation should be checked for more info.

More info

More info can be found at:

Extras

The LinOxide Team has created a Systemd vs SysV Init Linux Cheatsheet.

Related Question