Linux – I have high io stat. High writes. But what files are being written

ioslinux

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          11.50    0.02    5.38    0.07    0.00   83.04

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sdc               0.01    89.92    0.26   41.59     3.36   457.19    22.01     0.23    5.60   0.09   0.38
sdb               0.10    15.59    0.40   14.55     8.96   120.57    17.33     0.04    2.91   0.07   0.11
sda               0.13    45.37    0.96    8.09    20.06   213.56    51.63     0.02    2.64   0.16   0.14
sde               0.01    31.83    0.09   11.34     0.94   103.56    18.29     0.04    3.52   0.14   0.16
sdd               0.01    48.01    0.13   19.81     1.58   202.16    20.44     0.11    5.62   0.13   0.25

Is there a way to know what files are being written? 457 kB/s

Also this other linux machine have this same problem.

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          20.50    0.00   46.48   20.74    0.00   12.28

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.17    11.61    0.99    3.51    36.65    59.43    42.70     0.10   23.20   3.84   1.73
sdb               0.55   224.18   24.30   97.45   246.48  1287.12    25.19     3.96   32.53   7.88  95.91
sdd               0.53   226.75   25.56   90.96   283.50  1271.69    26.69     3.43   29.44   8.22  95.75
sdc               0.00     1.76    0.28    0.06     4.73     7.26    70.41     0.00   12.00   2.12   0.07
dm-0              0.00     0.00    1.11   14.77    36.41    58.92    12.01     1.00   62.86   1.09   1.74
dm-1              0.00     0.00    0.04    0.12     0.17     0.49     8.00     0.00   21.79   2.47   0.04
dm-2              0.00     0.00    0.01    0.00     0.05     0.01     8.50     0.00    7.90   2.08   0.00

1200 write request per second for a server that host nothing

Best Answer

Well, you could try the following commands which worked for me in RHEL6:

1) Whatever device you see in "iostat" output performing more I/O, use it with fuser command as follows:

fuser -uvm device

2) You will get a list of processes with the user name causing more I/O. Select those PIDS and use it in the lsof command as follows:

lsof -p PID | more

3) You will get a list of files/directories along with the user performing maximum I/O.

Related Solutions

linux – Understanding High Memory and Low Memory on Linux

On a 32-bit architecture, the address space range for addressing RAM is:

0x00000000 - 0xffffffff

or 4'294'967'295 (4 GB).

The linux kernel splits that up 3/1 (could also be 2/2, or 1/3 ¹) into user space (high memory) and kernel space (low memory) respectively.

The user space range:

0x00000000 - 0xbfffffff

Every newly spawned user process gets an address (range) inside this area. User processes are generally untrusted and therefore are forbidden to access the kernel space. Further, they are considered non-urgent, as a general rule, the kernel tries to defer the allocation of memory to those processes.

The kernel space range:

0xc0000000 - 0xffffffff

A kernel processes gets its address (range) here. The kernel can directly access this 1 GB of addresses (well, not the full 1 GB, there are 128 MB reserved for high memory access).

Processes spawned in kernel space are trusted, urgent and assumed error-free, the memory request gets processed instantaneously.

Every kernel process can also access the user space range if it wishes to. And to achieve this, the kernel maps an address from the user space (the high memory) to its kernel space (the low memory), the 128 MB mentioned above are especially reserved for this.

¹ Whether the split is 3/1, 2/2, or 1/3 is controlled by the CONFIG_VMSPLIT_... option; you can probably check under /boot/config* to see which option was selected for your kernel.

Linux – Guarantees for Concurrent Writes into a Named Pipe

It depends on how much each process is writing (assuming your OS is POSIX-compliant in this regard). From write():

Write requests to a pipe or FIFO shall be handled in the same way as a regular file with the following exceptions:
[...]

Write requests of {PIPE_BUF} bytes or less shall not be interleaved with data from other processes doing writes on the same pipe. Writes of greater than {PIPE_BUF} bytes may have data interleaved, on arbitrary boundaries, with writes by other processes, whether or not the O_NONBLOCK flag of the file status flags is set.

Also in the Rationale section regarding pipes and FIFOs:

Atomic/non-atomic: A write is atomic if the whole amount written in one operation is not interleaved with data from any other process. This is useful when there are multiple writers sending data to a single reader. Applications need to know how large a write request can be expected to be performed atomically. This maximum is called {PIPE_BUF}. This volume of POSIX.1-2008 does not say whether write requests for more than {PIPE_BUF} bytes are atomic, but requires that writes of {PIPE_BUF} or fewer bytes shall be atomic.

The value if PIPE_BUF is defined by each implementation, but the minimum is 512 bytes (see limits.h). On Linux, it's 4096 bytes (see pipe(7)).

Best Answer

Related Solutions

linux – Understanding High Memory and Low Memory on Linux

Linux – Guarantees for Concurrent Writes into a Named Pipe

Related Question