Ssh – Debian Lenny Server Compromised – No Sign Of Intrusion Point

Securityssh

I have a server, it's outdated running Debian Lenny, and yes I know that is probably half the problem. It also has PHPMyAdmin and ProFTPd. Again, I get it, all bad signs.

But for the life of me, I cannot figure out how this user is logging in and adding files and executing commands.

They are able to start screen sessions, and type things like nano file.sh and then create a script and ./file.sh to execute it. Does this mean they have SSH access? I don't understand.

I check all of my log files, and nothing anywhere shows successful authentication. I check users, who, last, every little command I can type – nothing shows any signs of someone being logged in.

Every now and again, I notice they create new directories and the owner is 500 or 1XXX, but these accounts don't show up when I look for them.

Is there something I can do to figure out wtf is going on? We are going to wipe the server clean, don't get me wrong, but I'd like to know what happened exactly so I can avoid this sort of problem in the future.

I don't want any recommendations regarding "don't use phpmyadmin, old unsupported distros, ftp, etc.", on our new server we won't have anything insecure, and will use passworded SSH Auth keys, etc.

I just want a bit of insight on how I can know when the user is logged in, and where they logged in from. Granted, I'm probably not giving enough information, but maybe something will click for someone? Thanks.

Best Answer

Most scripted and manual break-ins do:

  • clean up log entries and similar traces of the break-in
  • install a rootkit, which allows entry to the system outside of default server programs
  • replace default programs (like ps, netstat, ls, etc.) with manipulated versions which hide any activity of the above mentioned rootkit (ie. ps won't show the running rootkit process)

Sometimes those attacks are faulty and do leave traces behind. But in any case: you cannot trust any diagnostic tools you have on the system.

If you want to play a bit around and learn you could:

  1. Install and run 'rkhunter' [*] for example, which checks for known rootkits, but you cannot trust the output without:

    • having run it at least once before the break-in happened
    • hoping that the attacker ignored a rkhunter install on the system (did not manipulate rkhunter itself)

  2. Boot from a rescue CD/USB

    • Mount the systems disks and look around with the binaries of the rescue system
    • comparing md5sums of binaries with the stock version.
    • load the system into a VM and inspect the network traffic

tl;dr: It is near impossible to find out the attack vector on such an open system. One way or another:

Please be responsible and take the system off the internet ASAP and set it up newly from scratch.

[*] or other IDS systems, there are many.

Related Question