FreeBSD Shebang Error – How to Resolve Shebang Errors in FreeBSD

shebangshell

I would like to put shebang #!/bin/sh -eufo pipefail in my script. But there're several things strange:

  1. The script would fail with that shebang in FreeBSD but not when run on MacOS
  2. on FreeBSd, the same shebang works when directly executed from command line (also /bin/sh).
>>> sh -eufo pipefail -c 'echo hi'  # this works
hi

>>> cat <<EOF > script                                                                                      
#!/bin/sh -eufo pipefail
echo hi
EOF

>>> chmod +x ./script 
>>> ./script  # this doesn't work on FreeBSD but works on MacOS
Illegal option -o ./script

>>> cat ./script 
#!/bin/sh -eufo pipefail
echo hi

>>> uname -a
FreeBS 11.3-RELEASE-p7

Best Answer

MacOS still retains the old FreeBSD behaviour from before 2005. In 2005, there was a major change in the way that the FreeBSD kernel handled #! at the start of an executable file passed to execve(), to bring it more into line with some other operating system kernels, including Linux and the NetBSD kernel.

Commentary in the NetBSD kernel source code tries to paint this as a universal:

 * collect the shell argument.  everything after the shell name
 * is passed as ONE argument; that's the correct (historical)
 * behaviour.
— kern/exec-script.c. NetBSD. lines 189 et seq..

It actually is not. Sven Mascheck did some testing about a decade ago and there are four basic behaviours, the AT&T Unix System 5 one having as much claim to being "correct historical" behaviour as the 4.2BSD one has:

  • Ignore the characters (before 4.2BSD and AT&T Unix System 5).
  • Pass the whole string in a single argument (4.2BSD, NetBSD, Linux and FreeBSD from 2005 onwards).
  • Split the string up by whitespace and pass it as multiple arguments (FreeBSD before 2005 and MacOS).
  • Split the string up by whitespace and pass just the first argument (AT&T Unix System 5 and Solaris)

I've only included the operating systems relevant to this answer in parentheses. M. Mascheck checked a lot more, as did Ahmon Dancy in discussion of FreeBSD Problem Report 16393. See the further reading for the full lists.

What brought things to a head in FreeBSD in 2005 was that, ironically, FreeBSD wasn't quite as simple as that. It had had a change introduced that was intended to make things written in popular books about Perl actually work: arguments were skipped after a comment character. The books had recommended things like: 

#!/bin/sh -- # -*- perl -*-
— Larry Wall, Tom Christiansen, Jon Orwant (2000). Programming Perl: 3rd Edition. O'Reilly Media. ISBN 9780596000271. p. 488.

PR 16393 in 2000 was a way of making the kernel handle executable Perl scripts, written in the way that Larry Wall no less had said would work. However, it broke other stuff and didn't completely work.

There was some back and forth on this. Finally, in 2005 the mechanism to make Larry Wall et al.'s idea work was moved out of the kernel, which was made to behave compatibly with Linux, NetBSD, and 4.2BSD (rather than Solaris and AT&T Unix System 5) and made the responsibility of sh.

The behaviour since 2005 has thus been that the shell gets three arguments, the second argument being the entire tail of the #! line, and invoking your script directly with execve() is effectively the same as invoking:

sh '-eufo pipefail' ./script

It should be fairly obvious why the Almquist shell (which is what sh is on FreeBSD) is thinking that ./script is the option argument for the -o option, and that it is treating the pipefail part as further single-letter options collected behind - (which it hasn't got around to processing yet).

An also obvious alternative is to have set -o pipefail as the first command in the script, as pointed out at https://unix.stackexchange.com/a/533418/5132 for the Bourne Again shell. This was only added to the FreeBSD Almquist shell in 2019, however and thus is only available in very recent versions of FreeBSD. (The Debian Almquist shell has not yet had it added, as of 2020.)

Further reading

Related Question