Linux – How to tame Linux responsiveness, memory, and paging

First question on overflow =)… +100 bounty. Couldn't think of something I really cared about until now:

I'm really fed up with the state of Linux desktop responsiveness, e.g. http://brainstorm.ubuntu.com/item/85/ — in situations with low free-RAM, or situations with high disk throughput, the system slows to a crawl; this is absolutely terrible for applications which require decent performace. Additionally, the UI is completely unresponsive. Compare this for example with OS X, where if an application is hogging resources, one can always option-click to Force Quit it, whereas in Linux I cannot even alt-tab or switch desktop, or even ctrl-alt-f1 to get a terminal — well I can, it just takes about 1-2 minutes per operation.

I use gkrellm so I can see the situation as it unfolds. Typically the memory utilization becomes pretty high, or the disk throughput jumps dramatically.

It's not bad hardware, with a 2.6GHz quad-core and 4GB of 800MHz DDR2 RAM (would have had 6GB, but due to a hardware incompatibility couldn't mix-and-match with old set). This problem may go away when I inevitable get more RAM, but I don't feel that's the heart of the problem. I even have two swap partitions on different disks.

I feel the problem is threefold:

• runaway programs that hog up massive amounts of memory — the law must be laid down for these programs, with limits on their
  • (e.g. tabs on Chrome, each of which is 20-50MB, some of which can use hundreds of MB)
  • (e.g. other programs like update-db and indexers which I've had to disable and remove from cron because they were slowing the system to a crawl whenever they ran, etc.)
• something terrible going in the kernel or bus contention of some sort, such that high-disk-throughput situations slow the entire system to a crawl (perhaps by paging out important programs)
• the kernel is not prioritizing UI or important programs in terms of resources, such as memory, paging, even processor utilization

Upvotes go to:

I am thus looking for a solution where all such programs go away. In particular, I am looking for a solution such that the processes will slow down proportionally, while the system and other programs remains entirely unaffected and responsive long enough to manually kill something. Also the window manager process (and anything else that might affect UI responsiveness) should be responsive under all circumstances.

In particular I am intrigued by /etc/security/limits.conf (man limits.conf), but am worried this only gives per-user control, and the commented examples in the file seem rather opaque in terms of description or where to begin. I'm hoping that a limits.conf works, but would not be surprised if it didn't even work, or if it was not an appropriate solution for my problem, or as granular as I'm trying to achieve. A per-process-name limits.conf would be ideal, assuming again that limits.conf works. I'd be happy to try out a limits.conf that people provide, to test if it works, though I'm open to all solutions at this point.

It might also be useful to have any insights on how OS X manages to keep up such good UI responsiveness.

I have already tweaked my /tmp and cache folders to be on tmpfs, and in general disk utilization is near-zero.

Vaguely-related topics:

• memory overcommit

Answers I do not think will work:

• swapoff (this still lets memory hog programs get away with murder, and the system permanently freezing if memory is really bad — upvotes to anyone who can suggest a tweak that invoked the OOM-killer earlier before swapping and targets specific programs)
• echo ?? > /sys/.../swappiness (no discernable effect)
• nice (has never worked)
• ionice (never noticed a difference)
• selinux (program incompatibility seems to be a nightmare)
• realtime linux, i.e. can interrupt kernel (don't want to deal with compiling and updating custom kernel; might be okay if it has migrated into repositories)

*