Reserve resources for admin console

cgroupsload

My server is often almost DOS'ed by crawlers. I have not yet managed to figure out what exactly is responsible for the high load. CPU, memory and disk IO are not at the limit (could be network traffic). My question is, if anyone has an idea how to reserve capacity in the system to run a shell session (bash over ssh). Can this be done using cgroups? Right now load values easily go up to 60 and that make it hard to enter any command.

Best Answer

Something more basic: start sshd on a specific port with a very low nice level, which might help.

# nice -20 /usr/sbin/sshd -p 2222

IF the problem is due to DOS throttling the network, then you have to use network-shaping rules outside the box (ie, on the switch).

Related Solutions

Kernel – Why is kworker Consuming Many Resources on Linux 3.0.0-12-server?

I found this thread on lkml that answers your question a little. (It seems even Linus himself was puzzled as to how to find out the origin of those threads.)

Basically, there are two ways of doing this:

$ echo workqueue:workqueue_queue_work > /sys/kernel/debug/tracing/set_event
$ cat /sys/kernel/debug/tracing/trace_pipe > out.txt
(wait a few secs)

For this you will need ftrace to be compiled in your kernel, and to enable it with:

mount -t debugfs nodev /sys/kernel/debug

More information on the function tracer facilities of Linux is available in the ftrace.txt documentation.

This will output what threads are all doing, and is useful for tracing multiple small jobs.

cat /proc/THE_OFFENDING_KWORKER/stack

This will output the stack of a single thread doing a lot of work. It may allow you to find out what caused this specific thread to hog the CPU (for example). THE_OFFENDING_KWORKER is the pid of the kworker in the process list.

Load – How to Find Source of System Load for a Certain Period

I suggest to use atop. It's a daemon gathering all 'top' information every 10 minutes by default and you can just go back in time viewing these 'top' snapshots. Adjust the default interval setting to your needs (consumes more disk space if set more frequently).

Just yesterday, I answered a similar question, in which I included a very short how-to.

Best Answer

Related Solutions

Kernel – Why is kworker Consuming Many Resources on Linux 3.0.0-12-server?

Load – How to Find Source of System Load for a Certain Period

Related Question