bash – How to Make Reading and Writing the Same File in Pipeline Fail

bashio-redirectionpipeprocess

Say I have the following script:

#!/bin/bash
for i in $(seq 1000)
do
    cp /etc/passwd tmp
    cat tmp | head -1 | head -1 | head -1 > tmp  #this is the key line
    cat tmp
done

On the key line, I read and write the same file tmp which sometimes fails.

(I read it is because of race conditions because the processes in the pipeline are executed in parallel, which I do not understand why – each head needs to take the data from the previous one, doesn't it? This is NOT my main question, but you can answer it as well.)

When I run the script, it outputs about 200 lines. Is there any way I can force this script to output always 0 lines (so the I/O redirection to tmp is always prepared first and so the data is always destroyed)? To be clear, I mean changing the system settings, not this script.

Thanks for your ideas.

Best Answer

Gilles' answer explains the race condition. I'm just going to answer this part:

Is there any way I can force this script to output always 0 lines (so the I/O redirection to tmp is always prepared first and so the data is always destroyed)? To be clear, I mean changing the system settings

IDK if a tool for this already exists, but I have an idea for how one could be implemented. (But note this wouldn't be always 0 lines, just a useful tester that catches simple races like this easily, and some more complicated races. See @Gilles' comment.) It wouldn't guarantee that a script was safe, but might be a useful tool in testing, similar to testing a multi-threaded program on different CPUs, including weakly-ordered non-x86 CPUs like ARM.

You'd run it as racechecker bash foo.sh

Use the same system-call tracing / intercepting facilities that strace -f and ltrace -f use to attach to every child process. (On Linux, this is the same ptrace system call used by GDB and other debuggers to set breakpoints, single step, and modify memory / registers of another process.)

Instrument the open and openat system calls: when any process running under this tool makes a an open(2) system call (or openat) with O_RDONLY, sleep for maybe 1/2 or 1 second. Let other open system calls (especially ones including O_TRUNC) execute without delay.

This should allow the writer to win the race in nearly every race condition, unless system load was also high, or it was a complicated race condition where the truncation didn't happen until after some other read. So random variation of which open()s (and maybe read()s or writes) are delayed would increase the detection power of this tool, but of course without testing for an infinite amount of time with a delay simulator that will eventually cover all possible situations you can encounter in the real world, you can't be sure your scripts are free from races unless you read them carefully and prove they're not.

You would probably need it to whitelist (not delay open) for files in /usr/bin and /usr/lib so process-startup doesn't take forever. (Runtime dynamic linking has to open() multiple files (look at strace -eopen /bin/true or /bin/ls sometime), although if the parent shell itself is doing the truncation, that will be ok. But it will still be good for this tool to not make scripts unreasonably slow).

Or maybe whitelist every file the calling process doesn't have permission to truncate in the first place. i.e. the tracing process can make an access(2) system call before actually suspending the process that wanted to open() a file.

racechecker itself would have to be written in C, not in shell, but could maybe use strace's code as a starting point and might not take much work to implement.

You could maybe get the same functionality with a FUSE filesystem. There's probably a FUSE example of a pure passthrough filesystem, so you could add checks to the open() function in that which make it sleep for read-only opens but let truncation happen right away.

Related Question