Linux Ubuntu 12.04 high Load without high cpu/ram usage. Which tools and techniques to diagnose/solve this

From the memory usage related lines in top:

Mem: 16432032k total, 16340144k used, 91888k free, 21736k buffers
Swap: 18481144k total, 1112k used, 18480032k free, 15624488k cached

Let's ignore the swap. Total memory equals the sum of used and free memory. Used, on the other hand, is the sum of "really used by applications" and cached and buffers. So, in your case goes like this:

• Mem = 16432032k = 16340144k + 91888k;
• "Really used by applications" = Used - (cached + buffers) = 16340144k - (15624488k + 21736k) = 693920k.

The other 15.5 GB are cached. This improves performance. However, in the very moment an application requires part of the cached memory it is immediately given to it. You will notice this if you run some memory hungry application and monitor top.

CPU utilisation is easy:

From the command line:
while ( sleep 10 ) ; do cat /proc/loadavg >> mylogfile ; done

The sleep command will sleep 10 seconds and than return with the return value 0 (aka success). We abuse that to get a compact while( true ) sleep 10.

/proc/loadavg contains the load avarages of now, over the last 5 minutes, and over the last 15 minutes. If you are logging every 10 seconds then you are only interested in the first value.

Or in a script (using bash).

#!/bin/sh
# Using /bin/sh which is guaranteed to be present on any posix system.
# If you want to add shell specific parts in the script than replace this.
# E.g. if you want to use bash specific stuff then change it to:
# #!/usr/bin/env bash
# Make sure that the shebang is on the first line of the script (no comments above it!)

# While true, pause 10 seconds, then append information to Mylogfile
#
while ( sleep 10 ) ; do cat /proc/loadavg >> mylogfile ; done

We can add a cat /proc/meminfo to the information we append to the log file. /proc/meminfo is quite extensive and it will log a lot. If you only want to filter on specific memory information then please add that to the post.

The simplest form of that would result in:
while (sleep 10) ; do cat /proc/loadavg /proc/meminfo >> mylogfile ; done).