Ssh – How to find the cause of an ssh-scan exploit

forensicsssh

One customer emailed my about their provider about to shut down a server because ssh attacks.

I logged in to the server and found a lot of ssh-scan processes, and some strange files in /tmp.
I had to shut it down, since I didn't know what to do. Before rebuilding the server, is there a way I can find out how this happened in order to prevent it from happening again?

Best Answer

SSH scans are usually brute-force attacks. They just try common usernames with easy, common passwords. I've seen a system get compromised using the guest account, with password ‘guest’. Sigh.

Most machines are sprayed with such packets all the time. As a blanket solution, I like to do two things on the firewall:

Use geoip and ipset to allow access to port 22/tcp from specific countries only. There's a unusually high percentage of these attacks originating in .cn netblocks, for instance.
Use rate limiting on 22/tcp SYN packets so that the same IP address can only connect N times a minute before getting blocked for 10 or 15 minutes. This deters scanning software, and also slows down potential damage to other people's networks. It's a community service.

There are other ways too depending on your needs and restrictions.

On the target computers themselves, you should also lock down system accounts and implement a password policy that forbids easy passwords (wordlist checking, minimum length, etc).

Related Solutions

Ssh – How to prevent shutdown when an SSH user is logged in

I would write a small program that checked for any active SSH connections via netstat and/or ps. Drop it in place of the shutdown command.

If no one else is using the machine, call shutdown when the user tries to. If someone is using the machine, simply warn the user who issued the shutdown command.

Netstat will give you output like this, and it's pretty easy to look for .ssh in the output.

netstat -a
Active Internet connections (including servers)
Proto Recv-Q Send-Q  Local Address          Foreign Address        State
tcp        0     52  10.5.6.xx.ssh          10.6.6.yy.51400        ESTABLISHED
tcp        0      0  *.ssh                  *.*                    LISTEN
udp        0      0  *.syslog               *.*

ps will give you output like this, but it's a bit harder because you have to make sure not to worry about outbound connections. Netstat is probably the right way to go.

  ps -e | grep ssh
      10084366 ?        00:00:07 /opt/sbin/sshd
        282647 ?        00:00:00 /opt/sbin/sshd

Ssh – What audit log files are created in linux to track a users activities

As your system has been compromised, no information you get from that system can be trusted. Only logs that are immediately shipped off to an external system can be trusted (such as real time remote syslog). Meaning if you've got some nightly log rotation to an NFS share, you cannot trust it.

However it is possible the user did not bother covering his/her tracks, and you might have the information still available on the system.

Unfortunately on a default Centos/RHEL install, there is very minimal logging. You're basically restricted to poking around in /var/log. Which log you should dig through depends on what services are running on that box. However I would start with the ssh log. After that look at the logs of any services which run as root, or have sudo access.

If you're lucky, that test11 user might have a home directory, with a .bash_history file containing a history of what was done.

Also, as the system has been compromised to the point where an unknown user was able to gain root access, the system must be rebuilt from scratch. You cannot reuse anything from the system. Consider every single file as compromised. I would also recommend against using backups as you don't know how long ago the system was compromised.

Once a user gains root access, there are an unlimited number of backdoors that could be installed. If I were the one who had gained access to your system, simply removing that test11 user and changing the root password wouldn't even slow me down.

In the future, there are a few things you can do.

Remote logging

As mentioned, only real-time remote logging can be trusted not to be tampered with. Make sure you have this.

Auditing

There are 2 utilities you should install and use to monitor and audit the critical components of the system. These are auditd and ossec.

These 2 utilities operate differently, but are for the same goal, to watch for abnormal activity.

Terminal logging

There is another auditing tool called pam_tty_audit that works in conjunction with the earlier mentioned auditd utility. pam_tty_audit is a utility you add to your pam stack which logs all input & output across the TTY. Meaning if the user is accessing the box via interactive ssh, their activity would be logged.
Note however that it is of the utmost importance that this log be protected at all costs. This is mostly because of passwords. When the you type your password at a prompt, even though you don't see the password being typed, the pam_tty_audit module will see it, and log it. It is also possible you will cat out (or otherwise view) files containing sensitive information, which will also be logged. Thus this log must be either be immediately shipped off the local system so that it cannot be obtained by intruders, or it must be encrypted (and the decryption key must not be on the local system). Preferably both should be performed, ship off remotely, and encrypt it.