Performance Monitoring – Tools and Techniques

Since usbmon provides the length of each packet transferred, I would approach this by writing a quick program to parse the 0u file (which has data for all USB devices.) It would pick out the USB bus and device numbers, then keep a running total of the packet length field in both directions for each device.

This will then give you the amount of data transferred per device, in each direction. If you print it once a second you'll get a pretty good idea of each device's throughput. Note that it won't include any USB overhead, but if you compare the figures to a device that is able to saturate the available bandwidth you'll know whether you're getting close to the limit.

I managed to reproduce the problem again and it was result of a big disk cache. My disk caches can grow more than 8GB and seems that some applications doesn't like it and I/O suffers.

Dropping disk caches with echo 3 > /proc/sys/vm/drop_caches as root remedies the problem. I currently don't know why large disk caches causes this I/O degradation.

Last Update: After more investigation I've found out that number of files in the cache was triggering the problem. It was trashing the disks while trying to commit many small files back to the disk. Since I was using the system for ten years, I've took the plunge and reinstalled with 64 bit Debian. Now it's working smoothly. It was probably a side effect of ten years of upgrading with finding limits of 32 bit operating system.