Debian – How to pause (or capture) the messages that fly by at the end of startup sequence

debianstartupsystemd

Towards the end of the "startup sequence"1, I see a longish series of diagnostic messages fly by very quickly, right before I see the login prompt2.

AFAICT, most, if not all, of the lines making up this short-lived output begin with either of the strings shown below

[  OK  ]
[FAILED]

…where the OK is in green, and the FAILED is in red3.

These messages flash too briefly for me to read.

My question is:

Is there a way to make these messages easier to read?


Possible solutions that come to mind include (in order of preference):

  1. teeing (or simply redirecting) those messages verbatim4 to some persistent log file;
  2. enabling a paging-type mechanism (Press any key to continue...);
  3. inserting a pause (of configurable length) after those messages are printed;
  4. enabling some key (or key combination) to pause the output to the screen5.

EDIT: based on the comments I've gotten so far, I must conclude that the word verbatim in (1) above is either not being understood, or not being taken seriously, even though I have emphasized as much as I can. I would make it flash if I could…


EDIT2: the suggestion that meuh gave in the comments seems promising to me, but I have not been able to get it to work yet. Here's what I did:

First, I added the following at the end of /etc/rsyslog.conf:

# Save boot messages also to boot.log
local7.* /var/log/boot.log

…and rebooted. I saw the usual diagnostic messages fly by, but no /var/log/boot.log file was created.

Then, in the (admittedly unlikely) event that the /var/log/boot.log has to exist already before rsyslog can write to it, I executed (as root):

touch /var/log/boot.log
chgrp adm /var/log/boot.log
chmod 640 /var/log/boot.log

…where the chgrp and chmod commands were intended to make the ownership and permissions of /var/log/boot.log match those of all the other log files under /var/log. Then I rebooted, saw the messages, etc. The file /var/log/boot.log remained empty after this reboot.

(I got the same non-result when I changed the permissions of /var/log/boot.log to 666.)

I grep'ed the output of journalctl --boot and the files under /var/log for anything I could think of that may point to something awry with my rsyslog, but did not find anything. (I'm not at all familiar with rsyslog, so I'm sure my search was pretty inept.)

It's clear that what I've done so far is not sufficient to enable the desired logging. I'm now looking for whatever is that I'm missing. I have not been able to find much relevant documentation, though. For example, neither rsyslog.conf(5) nor rsyslogd(8) deigns to explain what local7 is (rsyslog.conf(5) is at least gracious enough to mention it once, without giving any further information).


EDIT3

Distro information:

$ lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 8.3 (jessie)
Release:    8.3
Codename:   jessie
$ uname -a
Linux myhost 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt20-1+deb8u3 (2016-01-17) x86_64 GNU/Linux

EDIT4

Additional potentially relevant information:

$ cat /lib/systemd/system/rsyslog.service
[Unit]
Description=System Logging Service
Requires=syslog.socket
Documentation=man:rsyslogd(8)
Documentation=http://www.rsyslog.com/doc/

[Service]
Type=notify
ExecStart=/usr/sbin/rsyslogd -n
StandardOutput=null
Restart=on-failure

[Install]
WantedBy=multi-user.target
Alias=syslog.service
$ cat /proc/$(pgrep rsyslogd)/limits
Limit                     Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            seconds   
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            8388608              unlimited            bytes     
Max core file size        0                    unlimited            bytes     
Max resident set          unlimited            unlimited            bytes     
Max processes             128529               128529               processes 
Max open files            1024                 4096                 files     
Max locked memory         65536                65536                bytes     
Max address space         unlimited            unlimited            bytes     
Max file locks            unlimited            unlimited            locks     
Max pending signals       128529               128529               signals   
Max msgqueue size         819200               819200               bytes     
Max nice priority         0                    0                    
Max realtime priority     0                    0                    
Max realtime timeout      unlimited            unlimited            us        

$ sudo ls /proc/$(pgrep rsyslogd)/fd | wc -l
10

1 I.e. that which takes place when I (re)boot my machine.

2 FWIW, multi-user.target is my default.

3 The remaining text is all in white over a black background. This is true of the subsequent login prompt.

4 I find completely unacceptable any solution that does not let me see the exact text of these messages as they appeared during the startup sequence. Since, invariably, I am not intimately familiar with whatever any of these diagnostic messages is referring to, it is not possible for me to recognize all the ways in which the underlying information conveyed by the original message may be paraphrased, spread over multiple other messages, subsumed in other messages, etc. (Only by searching online for the exact wording of the original message do I have any hope of finding a solution to the problem.) Everything I've tried so far, including journalctl -b and dmesg has failed at giving me the original messages verbatim. For example, when I run startup I can see only one red FAILED, but journalctl --boot | grep FAILED | wc -l returns 0, and journalctl --boot | grep -i FAILED | wc -l returns 1086. Neither of these is what I'm looking for.

5 In my system, I would have less than a second to press such key or key combination, and there is no forewarning of when this brief interval starts. Unless one is able to configure the duration of the interval during which such a keypress must happen, any key-press-based solutions is too impractical to be anything more than a last-resort maneuver. Also, FWIW, I did try pressing either the Scroll
Lock
or the Pause/
Break
key when the messages flashed, but neither made any difference.

Best Answer

You could set a kernel command line argument (something like console=tty0 console=ttyS0,115200n8) to send these to a serial console instead, and then the device that listens on the serial port can simply log it, since then it's just a stream of text.

And systemd is pretty silly if it doesn't log this stuff anyway. Openrc does it in /var/log/rc.log. Also if it wasn't systemd, you could probably modify inittab to just not put a getty/Xorg there on tty1, and prevent anything (like Xorg) from switching elsewhere, and the old text might stay put (just like it did on old pre-systemd openSUSE). Or copy it to another tty (which I think is syslog doing that rather than inittab... and you might see many linux installers doing this on tty9+) If it switches away and back, it just won't scroll back (shift+pgup), but will probably have one page of output. Perhaps someone that knows more about systemd knows the new equivalent to inittab and you can change that.

Related Question