SSH won’t work before manually restarting networking.service

debiannetworkingopensshssh

SOLVED!
The machine is a Dell Poweredge system which ran what they call iDRAC in the background – iDRAC in its turn ran a SSH-server which conflicted with the one I installed in Debian. The solution for me was just to disable iDRAC since I don't use it, and I'm glad I don't. I have no idea how that conflict should've been solved.

Just installed Debian Stretch on a machine. Minimal install haven't done pretty much anything but installing openssh-server and setting up my network interface. The problem I have seems to be related to the networking service.

Straight from boot I'm not able to connect to my server via SSH remotely. The server has internet connection at this time, I'm able to ping it and ping from it. At connection it starts out good but then a popup asks for a password for the user, typing in the password doesn't work – should already have been sent by the client anyway. When I run systemctl restart networking.service on the server SSH suddenly starts working.

openssh-server is running with default configurations.

My SSH client(Bitvise) log says:

15:08:26.493 Started a new SSH2 session.
15:08:26.493 Connecting to SSH2 server 171.xxx.xxx.xxx:22.
15:08:26.493 Connection established.
15:08:26.681 Server version: SSH-2.0-OpenSSH_5.8 <----
15:08:26.681 First key exchange started.
15:08:27.289 Received a new host key from the server. Algorithm: xxxxxxxxxxx
15:08:27.320 First key exchange completed using ecdh-sha2/nistp521. xxxxxxxxxxx
15:08:27.320 Attempting password authentication.
15:08:28.724 Authentication failed. Remaining authentication methods: 'password'.
15:08:30.581 Authentication aborted on user's request.
15:08:30.596 The SSH2 session has been terminated.

Note the server version SSH-2.0-OpenSSH_5.8

telnet 174.xxx.xxx.xxx 22

responds with SSH-2.0-OpenSSH_5.8

On the server SSH seems to be running just fine:

systemctl status sshd.service
● ssh.service - OpenBSD Secure Shell server
Loaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2017-06-28 11:16:52 CEST; 4h 10min ago
Main PID: 598 (sshd)
Tasks: 1 (limit: 7372)
CGroup: /system.slice/ssh.service
       └─ 598 /usr/sbin/sshd -D

systemd[1]: Starting OpenBSD Secure Shell server...
sshd[598]: Server listening on 0.0.0.0 port 22.
sshd[598]: Server listening on :: port 22.
systemd[1]: Started OpenBSD Secure Shell server.

Status for networking.service shows this

systemctl status networking.service
● networking.service - Raise network interfaces
Loaded: loaded (/lib/systemd/system/networking.service; enabled; vendor preset: enabled)
Active: active (exited) since Wed 2017-06-28 11:16:49 CEST; 4h 10min ago
  Docs: man:interfaces(5)
Process: 471 ExecStart=/sbin/ifup -a --read-environment (code=exited, status=0/SUCCESS)
Process: 335 ExecStartPre=/binsh -c [ "$CONFIGURE_INTERFACES" != "no" ] && [ -n "$(ifquery --read-environment --list --exclude=lo)" ] && udevadm settle (code=exited, status=0/SUCCESS)
Main PID: 1120 (code=exited, status=0/SUCCESS)
Tasks: 0 (limit: 7372)
CGroup: /system.slice/networking.service

systemd[1]: Starting Raise network interfaces...
systemd[1]: Started Raise network interfaces.

After restarting the network service

systemctl restart networking.service

After running this command SSH all of a sudden starts working.

SSH client log says:

15:08:57.179 Started a new SSH2 session.
15:08:57.179 Connecting to SSH2 server 171.xxx.xxx.xxx:22.
15:09:00.205 Connection established.
15:09:00.205 Server version: SSH-2.0-OpenSSH_7.4p1 Debian-10 <----
15:09:00.205 First key exchange started.
15:09:00.283 Received a new host key from the server. Algorithm: xxxxxxxxxxx
15:09:00.314 First key exchange completed using ecdh-sha2/nistp521. xxxxxxxxxxx
15:09:00.314 Attempting password authentication.
15:09:00.330 Authentication completed.
15:09:00.470 Terminal channel opened.

The odd thing here is that now it shows server version SSH-2.0-OpenSSH_7.4p1 Debian-10

telnet 174.xxx.xxx.xxx 22

responds with SSH-2.0-OpenSSH_7.4p1 Debian-10

systemctl status sshd.service shows same as before

Status for networking.service gives shows this

systemctl status networking.service
● networking.service - Raise network interfaces
Loaded: loaded (/lib/systemd/system/networking.service; enabled; vendor preset: enabled)
Active: active (exited) since Wed 2017-06-28 15:12:49 CEST; 1min 12s ago
  Docs: man:interfaces(5)
Main PID: 1292 (code=exited, status=0/SUCCESS)
Tasks: 0 (limit: 7372)
CGroup: /system.slice/networking.service

systemd[1]: Starting Raise network interfaces...
systemd[1]: Started Raise network interfaces.

/etc/network/interfaces

source /etc/network/interfaces.d/*

# The loopback network interface
auto lo
 iface lo inet loopback

auto eno1
 iface eno1 inet static
 address 174.xxx.xxx.29
 netmask 255.255.255.248
 gateway 174.xxx.xxx.25
 broadcast 174.xxx.xxx.31
 dns-nameservers 8.8.8.8 8.8.4.4

/etc/hosts

127.0.0.1 localhost.localdomain localhost
127.0.1.1 boris.secnet.sec boris

/etc/hostname

boris

/etc/resolv.conf

nameserver 8.8.8.8
nameserver 8.8.4.4

Made a resolv.conf and added nameservers there. Had issues with just having them in /etc/network/interfaces. Tried both with and without the resolv.conf and specifying them in /etc/network/interfaces.

NetworkManager enabled/disabled, no difference in behaviour.
No firewall.
The server is connected straight to the internet, same goes for my client machine.

So, what happens with SSH when systemctl restart network.services is invoked? Why does it throw back a different SSH server versions before and after the networking service is restarted?

Why won't the networking service do, whatever it does when restarting it manually, at boot?

I'm completely lost on this one, been testing for days. Worked perfectly fine with Debian Wheezy on the same net and interface settings before I upgraded. Made a full reinstall, formatted disk and installed Stretch.

Does anyone have a clue about what the problem might be? Or how to troubleshoot.

Best Answer

SOLVED! The machine is a Dell Poweredge system which ran what they call iDRAC in the background - iDRAC in its turn ran a SSH-server which conflicted with the one I installed in Debian. The solution for me was just to disable iDRAC since I don't use it, and I'm glad I don't. I have no idea how that conflict should've been solved.

Related Solutions

Debian – Migrate socat init script to systemd

Just figured out that I have to use

Type=forking

like described in http://www.freedesktop.org/software/systemd/man/systemd.service.html.

If set to forking, it is expected that the process configured with ExecStart= will call fork() as part of its start-up. The parent process is expected to exit when start-up is complete and all communication channels are set up. The child continues to run as the main daemon process. This is the behavior of traditional UNIX daemons. If this setting is used, it is recommended to also use the PIDFile= option, so that systemd can identify the main process of the daemon. systemd will proceed with starting follow-up units as soon as the parent process exits.

Debian – Failed to start MariaDB database server after upgrade to debian 9

In the tutorial you should install mariadb-server in place of mysql-server. I've got the same problem on DEBIAN 9 install and these command worked for me :

sudo apt purge mysql*
sudo apt purge mariadb*
sudo apt autoremove
sudo apt autoclean

Check if still something in the system :

dpkg -l | grep -e mysql -e mariadb

if the list is empty then install mariadb server :

sudo apt install mariadb-server

Now try to check status on stop and start the database :

Status after install :

sudo systemctl status mysql.service
● mariadb.service - MariaDB database server
   Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2018-03-21 16:08:49 CET; 2min 1s ago
 Main PID: 15699 (mysqld)
   Status: "Taking your SQL requests now..."
   CGroup: /system.slice/mariadb.service
           └─15699 /usr/sbin/mysqld

.../..

Check the PID :

sudo more /var/run/mysqld/mysqld.pid 
15699

Stop the database :

sudo systemctl stop mysql.service
sudo systemctl status mysql.service
● mariadb.service - MariaDB database server
   Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
   Active: inactive (dead) since Wed 2018-03-21 16:14:35 CET; 1s ago
 Main PID: 15699 (code=exited, status=0/SUCCESS)
   Status: "MariaDB server is down"

.../...

You could check the PID file is not present as well

sudo more /var/run/mysqld/mysqld.pid 
more: stat on /var/run/mysqld/mysqld.pid: No file or folder

Start the database :

sudo systemctl start mysql.service
sudo systemctl status mysql.service
● mariadb.service - MariaDB database server
   Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2018-03-21 16:14:54 CET; 3s ago
  Process: 15911 ExecStartPost=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
  Process: 15908 ExecStartPost=/etc/mysql/debian-start (code=exited, status=0/SUCCESS)
  Process: 15791 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= ||   VAR=`/usr/bin/galera_recovery`; [ $? -eq 0 ]   && systemctl set-environment _WSREP_S
  Process: 15787 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
  Process: 15784 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)
 Main PID: 15881 (mysqld)
   Status: "Taking your SQL requests now..."
    Tasks: 26 (limit: 4915)
   CGroup: /system.slice/mariadb.service
           └─15881 /usr/sbin/mysqld

You could check the PID file it should be present. May help on a empty database. If there is something in the database you should do a backup first.

Best Answer

Related Solutions

Debian – Migrate socat init script to systemd

Debian – Failed to start MariaDB database server after upgrade to debian 9

Related Question