IO – How to Limit Output Speed of stdout

hard-diskiostdout

I'm running CentOS 5.7 and I have a backup utility that has the option of dumping its backup file to stdout. The backup file is rather large (multiple gigabytes). The target is an SSHFS filesystem. To ensure that I don't hog the bandwidth and degrade the performance of the network, I would like to limit the speed with which data is written to the "disk".

How can I limit the ability of stdout based on a byte number? For example, limiting a process's ability to write to about 768Bps.

Best Answer

You can add a rate limiting tool to your pipeline.

For example there is pv which has a rate-limiting option:

-L RATE, --rate-limit RATE
Limit the transfer to a maximum of RATE bytes per second. A suffix of "k", "m", "g", or "t" can be added to denote kilobytes (*1024), megabytes, and so on

An alternative is the tool buffer which has:

   -u microseconds
After every write pause for this many microseconds. Defaults to zero. (Surprisingly a small sleep, 100 usecs, after each write can greatly enhance throughput on some drives.)

Related Solutions

Bash – copy consumed output to stdout when stdout is redirected to a file

tee the stdout stream to the controlling terminal device /dev/tty.

(
exec 1> >(tee -a stdout.log)
: > stdout.log
#var="$(echo -e "one\ntwo" | tee /dev/tty | grep one)"
var="$(echo -e "one\ntwo" | tee -a /dev/tty stdout.log | grep one)"
echo "var=$var"
)

cat stdout.log
# one
# two
# var=one

Cat – Command to Output File Content to Stdout

The most obvious one is cat. But, also have a look at head and tail. There are also other shell utillities to print a file line by line: sed, awk, grep. But those are to alternate the file content or to search inside the file.

I made a few tests to estimate which is the most effective one. I run all trough strace to see which made the least system calls. My file has 1275 lines.

awk: 1355 system calls
cat: 51 system calls
grep: 1337 system calls
head: 93 system calls
tail: 130 system calls
sed: 1378 system calls

As you can see, even if cat was designed to concatenate files, it is the fastest and most effective one. sed, awk and grep printed the file line by line, that why they have more that 1275 system calls.

Best Answer

Related Solutions

Bash – copy consumed output to stdout when stdout is redirected to a file

Cat – Command to Output File Content to Stdout

Related Question