Bash – Why does bash replaces text from command substitution with text thereafter

bashcatcommand-substitutionechofor

I was trying to parse some nginx config

λ tree sites-enabled/ sites-available/
sites-enabled/
├── bank.cwrcoding.com.conf
├── calendar.cwrcoding.com.conf
├── cloud.cwrcoding.com.conf
├── cwrcoding.com.conf
├── drive.cwrcoding.com.conf
├── groups.cwrcoding.com.conf
├── mail.cwrcoding.com.conf
├── sites.cwrcoding.com.conf
├── studentenverwaltung.cwrcoding.com.conf
├── wekan.cwrcoding.com.conf
└── www.cwrcoding.com.conf
sites-available/
├── bank.cwrcoding.com.conf
├── calendar.cwrcoding.com.conf
├── cloud.cwrcoding.com.conf
├── cwrcoding.com.conf
├── drive.cwrcoding.com.conf
├── groups.cwrcoding.com.conf
├── mail.cwrcoding.com.conf
├── sites.cwrcoding.com.conf
├── studentenverwaltung.cwrcoding.com.conf
├── wekan.cwrcoding.com.conf
└── www.cwrcoding.com.conf

The sites-enabled/* files each contain a single line:

include sites-availabe/cwrcoding.com.conf;

When trying to iterate over the sites-enabled/* files, cutting those and trying to read their contents as files, I got some weird error, so I tried a minimalistic working solution, and working my way up from there, but yet the following happens:

λ for enabled in sites-enabled/*
> do
> echo "$(cat "$enabled") |"
> echo ==========
> done
include sites-available/bank.cwrcoding.com.conf; |
==========
 |clude sites-available/calendar.cwrcoding.com.conf;
==========
 |clude sites-available/cloud.cwrcoding.com.conf;
==========
 |clude sites-available/cwrcoding.com.conf;
==========
 |clude sites-available/drive.cwrcoding.com.conf;
==========
 |clude sites-available/groups.cwrcoding.com.conf;
==========
 |clude sites-available/mail.cwrcoding.com.conf;
==========
 |clude sites-available/sites.cwrcoding.com.conf;
==========
 |clude sites-available/studentenverwaltung.cwrcoding.com.conf;
==========
include sites-available/wekan.cwrcoding.com.conf; |
==========
 |clude sites-available/www.cwrcoding.com.conf;
==========

As you can see, for most of the sites the first characters of the cat output are replaced by the text supposed to be after the command substitution.

Can anyone explain what is happening? Or have I found some bug?

If you want to take a look at the files: https://github.com/cwrau/nginx-config

Best Answer

The problem is that your files have DOS/Windows-style line-endings. As a quick work-around, replace:

echo "$(cat "$enabled") |"

With:

echo "$(tr -d '\r' <"$enabled") |"

Here, tr removes the carriage-return character before the file is displayed, avoiding the problem.

If your files are intended to be used on a Unix system, however, you would be better off removing the carriage-returns from the files themselves using one the dos2unix or similar utilities.

Example

Let's create two DOS-style files:

$ echo 'include sites-availabe/cwrcoding.com.conf;' | unix2dos > sites-enabled/file1
$ echo 'include sites-availabe/cwrcoding.com.conf;' | unix2dos > sites-enabled/file2

Let's run the original command:

$ for enabled in sites-enabled/*; do echo "$(cat "$enabled") |"; echo ==========; done
 |clude sites-availabe/cwrcoding.com.conf;
==========
 |clude sites-availabe/cwrcoding.com.conf;
==========

Note the mangled output.

With tr applied, we receive the output that we expect:

$ for enabled in sites-enabled/*; do echo "$(tr -d '\r' <"$enabled") |"; echo ==========; done
include sites-availabe/cwrcoding.com.conf; |
==========
include sites-availabe/cwrcoding.com.conf; |
==========

Related Solutions

Bash – Command substitution: cat with executable content

You want to do echo "$A". Wrapping the variable in the quotes makes it a string.

Example:

[root@talara test]# A=$(<test)
[root@talara test]# echo $A
FILE1 ham test test FILE1 ham test
[root@talara test]# echo "$A"
** test **

Bash: command substitution with multiple unset/export commands

In:

$(echo "
  export TEST_A=1
  export TEST_B=2
")

being $(...), just one expansion is parsed as a simple command.

So that $(...) expansion undergoes the split+glob operator. The output of echo is trimmed of its trailing newline characters and split into several words on IFS characters (newline, space and tab by default):

export
TEST_A=1
export
TEST_B=2

And each word undergoes globbing. Here they expand to themselves since they don't contain wildcard characters. So after globbing, we still have the same list of four strings.

The command to run is derived from the first one. Here export and all those strings are passed as arguments to that command. So, here, it's as if you had written:

export TEST_A=1 export TEST_B=2

So you're exporting the TEST_A variable with value 1, the export variable without changing its value and the TEST_B variable with value 2.

In the last one, you're running:

unset TEST_A export TEST_B=1

If you want the output of echo to be treated as shell code to interpret, that's where you want to use eval and not use the split+glob operator (quote your command substitution):

eval "$(echo "
  unset TEST_A
  export TEST_B=1
")"

That one is also a simple command. Two strings: eval and <newline><spaces>unset TEST_A<newline><spaces>export TEST_B=1.

eval evaluates the content of its arguments as shell code. So it runs those two shell command lines: unset TEST_A and export TEST_B=1 (again, two simple commands).

Best Answer

Example

Related Solutions

Bash – Command substitution: cat with executable content

Bash: command substitution with multiple unset/export commands

Related Question