Shell – Make mistakenly thinks rule succeeded because of 0-sized file generated by output redirect

io-redirectionmakeshell

In a makefile, I have several rules that look like this:

out.txt: foo.sh input.txt
  ./foo.sh -i input.txt > out.txt

If foo.sh fails, then out.txt will be created as a 0-sized file. If I run make again, it will mistakenly assume that the out.txt file was created successfully, and it won't run the rule again.

What's the right way to handle this sort of scenario?

Best Answer

You can request that make delete the target file if the rule fails, by defining a special target named .DELETE_ON_ERROR. It doesn't need to do anything or have any dependencies, so just add this to your makefile:

.DELETE_ON_ERROR:

Then you get the following:

$ cat Makefile
.DELETE_ON_ERROR:
foo:
    false > foo

$ make
false > foo
make: *** [foo] Error 1
make: *** Deleting file `foo'
zsh: exit 2     make

$ stat foo
stat: cannot stat `foo': No such file or directory

From Errors in Recipes:

Usually when a recipe line fails, if it has changed the target file at all, the file is corrupted and cannot be used—or at least it is not completely updated. Yet the file's time stamp says that it is now up to date, so the next time make runs, it will not try to update that file. The situation is just the same as when the shell is killed by a signal; see Interrupts. So generally the right thing to do is to delete the target file if the recipe fails after beginning to change the file. make will do this if .DELETE_ON_ERROR appears as a target. This is almost always what you want make to do, but it is not historical practice; so for compatibility, you must explicitly request it.

Related Solutions

Shell – Use HERE file and redirect output of command

What you've written in the first line looks like a complete command (a “(compound) list” in shell terminology), so the shell treats it as a complete command. Since there's a here-document start marker (<<END), the shell then reads the here-document contents, and then starts a new command. If you want to put the here-document in the middle of a list, you need to indicate to the shell that the list is not finished. Here are couple of ways.

mpirun -np 6 ./laplace <<END |
…
END
tail -n 1 > output

{ mpirun -np 6 ./laplace <<END
…
END
} | tail -n 1 > output

Or, of course, you can make sure the command completely fits in the first line.

mpirun -np 6 ./laplace <<END | tail -n 1 > output
…
END

The rule to remember is that the here-document contents starts after the first unquoted newline after the <<END indicator. For example, here's another obfuscated way of writing this script:

mpirun -np 6 ./laplace <<END \
| tail -n $(
…
END
             echo 1) > output

Shell IO Redirection – Redirect Output from All Commands to File, Not Just Last

The problem here isn't an issue with Linux redirection; rather, it's a fundamental misunderstanding of how the pipeline works. Redirection here isn't working because only cut is actually printing to stdout. stdout for the echo command has been piped to cut's stdin (which isn't being used in this case since a file is specified).

echo "header line" > output_file && cut -c 1-5 input_file >> output_file

is what you want, and not inelegant at all (I replaced your ; with && so that the cut command will only execute if the header is successfully written; this way it won't execute if you don't have permissions to create or write to output_file).

You could also do it all in a subshell, eg.

(echo "header line"; cut -c 1-5 input_file) > output_file

but there's no real benefit to doing this and with more complex examples it can cause issues if you're not familiar with how the subshell is scoped.

If you want cut to pass stdin through to stdout you could try:

echo "header line" | cut -c 1-5 - input_file

(The dash is a common shortcut for stdin)

However, this will also perform the cut operation on stdin (resulting in a header line of "header"). It's hard to tell if this is what you want or not from the question.

Best Answer

Related Solutions

Shell – Use HERE file and redirect output of command

Shell IO Redirection – Redirect Output from All Commands to File, Not Just Last

Related Question