bash – Performing Atomic Write Operations in a File

bashcommand linelinuxsemaphore

After going through the bash documentation, this question and this one it's still not clear to me how can I perform atomic write (append) operations to a file in bash. I have a script that is run in multiple instances and at some point must write data to a file:

echo "$RESULT" >> `pwd`/$TEMP_DIR/$OUT_FILE

How is it possible to make all write operations from all concurrently running scripts to that file atomic (so that data from one instance doesn't overlap data from another)?

Best Answer

It seems you need to use flock as in the example from man (http://linux.die.net/man/1/flock)

(
flock -x 200

# Put here your commands that must do some writes atomically

) 200>/var/lock/mylockfile

And put all your commands that must be atomic in ().

Related Solutions

Bash Command – Printing and Saving the Last Bash Input Command

You can access the just-executed command line with the history built-in. (I have no idea why history 1 prints the just-executed command line but fc -nl -1 prints the previous commmand, as does fc -nl 0.)

PROMPT_COMMAND='echo -en "\033[38;5;2m"; history 1; echo -en "\033[0m\n"'

This prints a number before the command text. Here's a version that removes the number. (It may be incorrect if you go beyond 99999 history lines, I don't know how bash formats the number then.)

prompt_function () {
  local prompt_history="$(history 1)"
  prompt_history=${prompt_history:7}
  echo -En $'\033[38;5;2m'"$prompt_history"$'\033[0m\n'
}
PROMPT_COMMAND=prompt_function

(Note that echo -en ..."$prompt_history"... would expand backslashes in the command line, so I use echo -E and let the shell expansion generate the control characters with $''.).

How to Use Flock and File Descriptors to Lock a File

This is not correct because when you do ( flock -e 200; ... ) 200> file, you are truncating the file file before you get the exclusive lock. I think that you should do:

touch resp.txt
(
  flock -e 200
  curl 'someurl' -H 'someHeader' > resp.txt
) 200< resp.txt

to place the lock on the file opened as read only.

Note. Some shells do not support file descriptors larger than 9. Moreover the hardcoded file descriptor may already be used. With advanced shells (bash, ksh93, zsh), the following can be done:

touch resp.txt
(
  unset foo
  exec {foo}< resp.txt
  flock -e $foo
  curl 'someurl' -H 'someHeader' > resp.txt
)

Best Answer

Related Solutions

Bash Command – Printing and Saving the Last Bash Input Command

How to Use Flock and File Descriptors to Lock a File

Related Question