3.x is just continuation of 2.x - at one point Linus decided that the "x" part of the version is too big. Generally you probably want reasonably recent kernel, probably one marked as "longterm". A lot also depends on your application as well - while remote security holes in kernel are rather scarce, local problems are much more prevalent.
You can also try this:
Use sar to report on context-switches and irq use over time. Sar is a great but unhearalded system monitoring tool. Run it for a day and then use various reports to look for oddness.
Steps:
Install the sysstat package. Set up the cron job (check /etc/cron.d/ to see if such a file exists) like this:
* * * * * root /usr/lib64/sa/sa1 -S XALL 10 6
This will create lots of data -- about 300 MB per day. It will take a snapshot of the system statistics every 6 seconds. (You can decrease the frequency to twice per minute by changing "10 6" to "2 30")
If you change the capturing frequency, you should erase the day's file in /var/log/sa/sa$(date +%d) -- otherwise the reports might get corrupted.
OK, after you have captured your data, try these (Tip: always pipe through less):
sar -q
To get load-averages and the run-queue sizes. If these are low, then look at the IO wait times:
sar -P ALL
If you don't see jumps in %iowait or troughs in %idle, then check out context-switches:
saw -w
500 context-switches per second is pretty normal on a mostly-idle system.
You can limit sar reporting by specifying the activity file, giving a start- and end-time (-s <hh:mm:ss> and -e <hh:mm:sss> respectively) and optionally an interval time with -i <secs> (must be greater than your capturing resolution, set up by the cron job).
Example:
sar -f /var/log/sa/sa$(date +%d -d yesterday) -s 09:15:00 -e 12:15:00 -i 600
will give you summary info for the CPU in 10-minute intervals from yesterday between 9:15 and 12:15.
There's more... much more. You can do man sar to get that info.
Best Answer
The kernel memory management doesn't use the concept of "program" to organize the memory, but pages. The kernel decides based on a 'least recently used' (this may have changed, but the latest references I could find say so), in which when the kernel under memory pressure (by swappiness, it swaps out those pages which are rarely accessed to give space to more frequently accessed or new pages (or why not, disk cache).
So, none of your assumptions are correct. The kernel doesn't order programs to the swap, it statistically select which parts of the memory will commit to swap. The method is LRU. The size of anonymous or private memory committed by the program doesn't affect the way the kernel will select which pages will be committed.
You can see how many KiloBytes (or pages) have the kernel committed to swap using
smem -s swap -t -k -n.