Linux – How to obtain counters for swap-in/swap-out on Linux

The difference you are observing isn't actually due to swap space being unaccounted for. The "(deleted)" that the kernel sometimes appends to /proc/*/exe links is output by readlink and is causing parse errors in your awk script, and you are effectively not counting processes whose binaries are no longer present in your total.

Some kernels append the word "(deleted)" to /proc/*/exe symlink targets when the original executable for the process is no longer around.

The reason your command is showing less than the total is because of this. The output of readlink on such links will be something like "/path/to/bin (deleted)", which causes a parse error in awk when the output is substituted back into the string (it doesn't like the parentheses and spaces). For example, do this:

for a in /proc/*/exe ; do readlink $a ; done | grep deleted

And you will see a few entries with "(deleted)" appended. If you looked at the swap usage for these entries, their total would match the discrepancy you see, as the resulting awk errors prevent their totals from being calculated and included in the final total.

If you run your original command without redirecting stderr anywhere, you will probably notice a few "runaway string constant" errors. Those errors are a result of the above and you should not have ignored them.

Ignoring other potential improvements to your original command, you could modify it by removing the " (deleted)", like this (note |awk '{print $1}' added to readlink output):

for proc in /proc/*; \
  do cat $proc/smaps 2>/dev/null | awk '/Swap/{swap+=$2}END{print swap "\t'`readlink $proc/exe|awk '{print $1}' `'" }'; \
done | sort -n | awk '{total+=$1}/[0-9]/;END{print total "\tTotal"}'

This use of awk to fix the output of readlink may break if the name contains spaces -- you can use sed or whatever method you prefer.

Bonus Info

By the way, you could just use smem -t. The "Swap" column displays what you want.

As for calculating it yourself, though, you can also get this information more directly from the VmSwap field in /proc/*/status (smaps requires some kernel support and isn't always available), and avoid having to redirect error output by using a proper filename pattern that avoids the errors to begin with:

for proc in /proc/[0-9]*; do \
  awk '/VmSwap/ { print $2 "\t'`readlink $proc/exe | awk '{ print $1 }'`'" }' $proc/status; \
done | sort -n | awk '{ total += $1 ; print $0 } END { print total "\tTotal" }'

If you don't need the actual binary and can deal with just having the process name, you can get everything from status:

for a in /proc/*/status ; do \
  awk '/VmSwap|Name/ { printf $2 " " } END { print "" }' $a ; \
done | awk '{ total+=$2 ; print $0 } END { print "Total " total }'

And finally, if just having the PIDs suffices, you can just do it all with awk:

awk '/VmSwap/ { total += $2; print $2 "\t" FILENAME } END { print total "\tTotal" }' /proc/*/status

Note:

Now this isn't to say that there aren't differences between free and smem (the latter being the same as your script). There are plenty (see, for example, https://www.google.com/search?q=smem+free, which has more than enough results on the first page to answer your questions about memory usage). But without a proper test, your specific situation cannot be addressed.

Here's the list of -graph options available in kSar v5.0.6 -

all-cpu
bond0-if1
bond0-if2
eth0-if1
eth0-if2
eth2-if1
eth2-if2
LinuxcswchSar
LinuxioSar
LinuxkbmemSar
LinuxkbmiscSar
LinuxkbswpSar
LinuxloadSar
LinuxnfsdSar
LinuxnfsSar
LinuxpageSar
LinuxpgpSar
LinuxprocSar
LinuxsockSar
LinuxswapSar
lo-if1
lo-if2

Here's an example of how I generate a report -

java -jar kSar.jar -input /apps/kSar-5.0.6/logs/sa01 -showCPUstacked -cpuFixedAxis -graph 'all-cpu bond0-if1 LinuxcswchSar LinuxioSar LinuxkbmemSar LinuxloadSar LinuxpgpSar LinuxprocSar'   -outputPDF /apps/kSar-5.0.6/logs/sa01.pdf