Ubuntu – What causes SSH problems after rebooting a 14.04 server

14.04opensshserverssh

Why does rebooting a server running Ubuntu 14.04 give me 'Connection refused' errors?

I see ssh: connect to host <IP-address-here> port 22: Connection refused but only for 14.04 and only after rebooting. I'm using 12.04 Desktop at home. How do I troubleshoot this?

To make the question clearer, here's what does or does not work for me:

  • SSH into a fresh install of 12.04 > logout > SSH in again > works
  • SSH into a fresh install of 12.04 > reboot > SSH in again > works
  • SSH into a fresh install of 14.04 > logout > SSH in again > works
  • SSH into a fresh install of 14.04 > reboot > SSH in again > Connection refused

The problem I'm having is unique to 14.04, and only happens after rebooting. I have several servers running 12.04 prior to this and everything still works perfectly. I've got a new server I want to use 14.04 on and I want to understand what is going wrong. Any suggestions?

Here's what I've tried so far:

sudo traceroute -p 22 -T <IP-address-here>

Traceroute works fine, I get a response from the server on SSH port 22.

initctl list
...
ssh start/running, process 23371
...

Looks like ssh on the 14.04 server is set to start at boot (as expected).

tom@Desktop:~$ ssh -vvv root@<IP-address-here>
OpenSSH_5.9p1 Debian-5ubuntu1.4, OpenSSL 1.0.1 14 Mar 2012
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to <IP-address-here> [<IP-address-here>] port 22.
debug1: connect to address <IP-address-here> port 22: Connection refused
ssh: connect to host <IP-address-here> port 22: Connection refused

Edit: Here is the entire syslog from a freshly created machine. I created it, SSH'd in & issued reboot now command, then got a connection refused error after waiting for it to reboot and trying to SSH in the second time. Hard reboot via hosting control panel and now SSH connection works again.

Best Answer

Quick answer:

SSH is not the problem. The command you use to reboot is the problem: don't do reboot now, do reboot or shutdown -r now to reboot your system.

The command syntax (since 13.04) has been:

reboot [OPTION]...  [REBOOTCOMMAND]

The REBOOTCOMMAND never existed before. In 12.04, your now was just ignored but now it's being used... And it's breaking everything.

Long answer, with my tests results and explanation:

I have a similar problem with some servers running 14.04 AND in VPS (hosted at the French OVH provider - running OpenVZ) AND when doing reboot now inside the server itself.

Like you I've issued the command reboot now from the console (logged in using SSH). A few second after I pressed RETURN, my session is automatically disconnected. Like you I've never been able to reconnect to the server via SSH after issuing this command.

So, I decided to open the KVM console provided by OVH. (emulating the direct access using keyboard and screen on a physical server for this kind of virtual server).

I was able to connect to my machine and I saw that she was entering into Single User Mode, waiting for me to press CTRL+D to continue or to enter the root password to go into maintenance mode. I pressed the key combination to go let the process continue and then was able to SSH into my system again. What was my surpise to see, after running uptime, that the uptime was not 2 or 3 minutes but yet a lot of day : reboot now executed inside an Ubuntu 14.04 VPS is not really rebooting but is just asking to go into Single User mode!

From this, I've learned to never ask a reboot from within my VPS but to request it from the command provided on the management interface of the hoster.

Thus there is no problem with your SSH installation. The problem is when you type reboot now. In fact, I tested it afterward also, if you had typed reboot (just the word, no option), it would have done what you were intending to do : reboot the server.

Using reboot with an argument (from the man page) call the command shutdown with the given arguments. And indeed, if I execute shutdown now, I have the same behaviour : the system is not rebooted, it goes into single user mode.

Remark: it looks like it is the intended behaviour as the message appearing on the screen after hiting executing this command says something like :

The system will be brought to the maintenance mode

Maintenance mode or single user mode, this represent the same, a runlevel with noting more than a shell, no network, no network processes, ...

This may be confusing, but note that the correct usage of shutdown is, for instance : shutdown -h now to halt the system now or shutdown -r now to reboot it now. I wasn't aware that shutdown now would only bring the system into single user mode. I usually do init S to achieve this.

Related Question