Linux – How to Monitor Disk IO

You can use the tool iostat to collect the disk utilization information. It takes several arguments, including the switch, -d:

   -d     Display the device utilization report.

It also takes an argument in seconds an interval of how frequent it should re-run. The value 3600 would be the number of seconds in an hour.

Example

$ iostat -d 3600
Linux 2.6.35.14-106.fc14.x86_64 (grinchy)   09/04/2013  _x86_64_    (4 CPU)

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda              20.53        71.86      1259.61   20308334  356000380
dm-0              4.92        39.02        28.81   11027610    8143376
dm-1              0.54         0.93         3.38     261472     954912
dm-2            156.65        31.87      1227.42    9006394  346902056

The output from this command could be redirected to a file:

$ iostat -d 3600 >> iostat_hrly.log

Meaning of the units

If you consult the man page for iostat it has pretty good descriptions of the units.

excerpt

   Blk_read/s
         Indicate the amount of data read from the device expressed in a 
         number of blocks per second. Blocks are equivalent to sectors with
         kernels 2.4 and later and  therefore have a size of 512 bytes. With
         older kernels, a block is of indeterminate size.

  Blk_wrtn/s
         Indicate the amount of data written to the device expressed in a 
         number of blocks per second.

  Blk_read
         The total number of blocks read.

  Blk_wrtn
         The total number of blocks written.

So a block is 512 bytes, so the Blk_read/s in terms of MB for device sda would be, 71.86 * 512 bytes = 36.79232 kilobytes/sec.

There are additional switches that will change the units automatically in the output.

excerpt from iostat man page

-h     Make the NFS report displayed by option -n easier to read by a human.

-k     Display statistics in kilobytes per second instead of blocks per 
       second.  Data displayed are valid only with kernels 2.4 and later.

-m     Display statistics in megabytes per second instead of blocks or 
       kilobytes per second.  Data displayed are valid only with kernels 
       2.4 and later.

Example in KB/s

So this might be more useful, showing the throughput in KB/s:

$ iostat -dk 3600
Linux 2.6.35.14-106.fc14.x86_64 (grinchy)   09/05/2013  _x86_64_    (4 CPU)

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda              20.85        47.25       663.81   15475096  217427086
dm-0              5.01        20.00        14.43    6549301    4725068
dm-1              0.54         0.58         1.60     189064     524872
dm-2            165.30        26.65       647.78    8730281  212177124

There are several ways to tune I/O performance.

• file system choice - choose a file system depending on what your typical data looks like - some file systems handle better lots of smaller files, some do well for large files, some don't scale well when accessing (especially creating) multiple files at once. Defragmenting might help on spinning plate drives (not so much on flash based devices, but the difference should still be there).

• file system tweaking - modern file systems have loads of options (both create- and mount-time) which push performance of the file system in one direction or another. Prime example can be how journalling is done.

• DMA setting - while this has been done automagically since ages, it never hurts to check it (hdparm -i is your friend here).

Linux specifically (but might be similar for other kernels as well):

• THP - Transparent Huge Pages used to cause performance degradation. IIRC the problem has been fixed (at least to some extent) some time ago. If copying to/from USB is the issue, you probably should look in this direction. disable transparent hugepages might serve as a good starting point as for how to disable THP.

• kernel I/O scheduler can influence the performance a big deal. Some people advocate the use of the Deadline scheduler over CFQ which is more complex and sometimes allegedly tends to overthink things.

Apart from that, in some scenarios heavy I/O just happens. When you start copying files from USB memory card reader to hard drive, from hard drive to a USB flash disk, browse the web (some browsers might be fsync() heavy to make sure things are recoverable in case of a crash), download emails, run a P2P client and play a movie all at the same time, the physical disk and the file system are just going to become the bottleneck. In some cases a carefully constructed (read with appropriate stripe size and number of constituent drives) RAID0 might help to alleviate the problem.