Shell – Why Does Command Substitution Remove Trailing Newline?

command-substitutionshelltext processing

As per the following example, and as in my recent question In bash, where has the trailing newline char gone?, I want to know "why" it happens

  x="$(echo -ne "a\nb\n")" ; echo -n "$x" | xxd -p 
# Output is: 610a62 
# The trailing newline from the 'echo' command
#   has been "deleted" by Command Substitution

I assume there must be some very significant reason for a shell action, namely Command Substitution, to actually delete some data from the command output it is substituting…
but I can't get my head around this one, as it seems to be the antithesis of what it is supposed to do.. ie. to pass the output of a command back into the script process… Holding back one character seems weird to me, but I suppose there is a sensible reason for it… I'm keen to find out what that reason is…

Best Answer

Because the shell was not originally intended to be a full programming language.

It is quite difficult to remove a trailing \n from some command output. However, for display purposes, almost all commands end their output with \n, so… there has to be a simple way to remove it when you want to use it in another command. Automatic removal with the $() construction was the chosen solution.

So, maybe you'll accept this question as an answer:

Can you find a simple way to remove the trailing \n if this was not done automatically in the following command?

> echo The current date is "$(date)", have a good day!

Note that quoting is required to prevent smashing of double spaces that may appear in formatted dates.

Related Solutions

Shell Command – Missing Trailing Newline Character in Command Substitution

The command substitution function $() (and its cousin the backtick) specifically removes trailing newlines. This is the documented behavior, and you should always be aware of it when using the construct.

Newlines inside the text body are not removed by the substitution operator, but they may also be removed when doing word splitting on the shell, so how that turns out depends on whether you used quotes or not. Note the difference between these two usages:

$ echo -n "$(echo -n 'a\nb')"
a
b

$ !! | xxd -p
610a62

$ echo -n  $(echo -n 'a\nb')
a b

$ !! | xxd -p   
612062

In the second example, the output wasn't quoted and the newline was interpreted as a word-split, making it show up in the output as a space!

Bash Here-String – Why It Adds a Trailing Newline Character

The easy answer is because ksh is written that way (and bash is compatible). But there's a reason for that design choice.

Most commands expect text input. In the unix world, a text file consists of a sequence of lines, each ending in a newline. So in most cases a final newline is required. An especially common case is to grab the output of a command with a command susbtitution, process it in some way, then pass it to another command. The command substitution strips final newlines; <<< puts one back.

tmp=$(foo)
tmp=${tmp//hello/world}
tmp=${tmp#prefix}
bar <<<$tmp

Bash and ksh can't manipulate binary data anyway (it can't cope with null characters), so it's not surprising that their facilities are geared towards text data.

The <<< here-string syntax is mostly only for convenience anyway, like << here-documents. If you need to not add a final newline, use echo -n (in bash) or printf and a pipeline.

Best Answer

Related Solutions

Shell Command – Missing Trailing Newline Character in Command Substitution

Bash Here-String – Why It Adds a Trailing Newline Character

Related Question