Occasionally, one VPS under my control gets bombarded with POP3 login attempts. They all fail, but the huge number of spawned processes and active connections pretty much amount to a DoS attack.
When I see that going on, I usually go in and manually block the offending IP address in iptables. What I would like is to have a process running on the server monitoring the system log (or some other appropriate place) and, if there are repeat log entries matching a particular pattern, pass them to a command which would then extract the relevant part (the remote host IP address, in this case) and execute a command (to add a DROP rule to iptables). For example, do so if log entries with the same message-part are logged five times within one minute.
The VPS runs syslog-ng in case that helps. I have set up rate limiting in iptables which helps somewhat, but certainly isn't perfect as it blocks my own connection attempts just as much as those of an attacker (it becomes the luck of the draw who gets an established connection). Since the clients that are supposed to connect have IP addresses from dynamic blocks, it's difficult to just add an override rule without rate limiting.
Since the VPS runs on Virtuozzo, while I do have root access to the guest, I cannot load custom kernel modules, or a custom kernel. So it has to be done in userspace.
What software will help me?
Best Answer
I'd recommend fail2ban.
Fail2ban is software which is meant to monitor logs for brute force login attempts. When it see's such an attempt, it blocks the attacker's IP via iptables. After a sufficient amount of time has passed fail2ban will automatically remove the block.
Fail2ban is customizable and can work with just about any type of internet service daemon. There is even specific documentation for pop3 daemons such as courier, dovecot, and qmail.