Bash – sed using double quotes with \!b leads to unknown command on back slash

bashquotingsed

I currently have a sed command that I want to act on the following type of text:

user:
        ensure: 'present'
        uid: '666'
        gid: '100'
        home: '/home/example'
        comment: ''
        password_max_age: '99999'
        password_min_age: '0'
        shell: '/bin/false'
        password: ''

I can get the type of results I want with this command:

sed '/user:/!b;n;n;n;n;n;n;n;n;n;s/.*/\t\tpassword: \x27\!\!\x27/g'

user:
        ensure: 'present'
        uid: '666'
        gid: '100'
        home: '/home/example'
        comment: ''
        password_max_age: '99999'
        password_min_age: '0'
        shell: '/bin/false'
        password: '!!'

The problem with that command is that user: is static. I want to be able to iterate through a list of users and use a bash variable instead of a specific user in the sed command. To do that though, I need to use double quotes instead of single quotes. However, when I use this command:

sed "/user:/!b;n;n;n;n;n;n;n;n;n;s/.*/\t\tpassword: \x27\!\!\x27/g"

It complains about the '!' in the !b sed command I'm using.(since bash is trying to interpret it) However, if I escape it like so:

sed "/user:/\!b;n;n;n;n;n;n;n;n;n;s/.*/\t\tpassword: \x27\!\!\x27/g"

Then I get this error:

sed: -e expression #1, char 6: unknown command: `\'

How can I get this to work?

Best Answer

Quoting does not delimit fields.

This is an important, but often forgotten, aspect of shell language syntax. The mental model that one "quotes arguments" is in fact simplistic and wrong. One quotes stuff that needs quoting, and that need only be part of what ends up as a single argument.

And that is exactly what you need to do here. Ironically, the first now-deleted answer was very close.

The final string that you want to give to sed as the actual single argument that it executes with is 

/user:/!b;n;n;n;n;n;n;n;n;n;s/.*/\t\tpassword: \x27!!\x27/g
with the user part varying according to the actual user name. But to get here you do not need to use one single quoting scheme for the entire argument. You can build up the argument from parts.

Use double quotes for the part where you need parameter expansion to occur, and single quotes for the parts where you need history expansion to not occur:

while read -r i
do
   sed -e '/'"$i"':/!b;n;n;n;n;n;n;n;n;n;s/.*/\t\tpassword: \x27!!\x27/g' puppet-users.yaml > puppet-users{new}.yaml
   mv puppet-users{new}.yaml puppet-users.yaml
done < user_list.txt

This is:

  1. The single-quoted character /.
  2. The double-quoted characters $i which are subject to parameter expansion (and of course all other expansions and substitutions).
  3. The single-quoted characters :/!b;n;n;n;n;n;n;n;n;n;s/.*/\t\tpassword: \x27!!\x27/g which are not subject to history expansion.

Field splitting only occurs on unquoted characters; and there are none such here at all, so this is all one field, which becomes all one argument to the sed command.

Once you've done this, you'll discover that you've used too many TAB characters in what you are doing. ☺

On the gripping hand …

sed is not quite the right tool for this job. Nor indeed is awk, as in the second now-deleted answer. Parsing YAML using a proper YAML parser, such as Python's yaml module, will be better in the long run, if the task at hand becomes less trivial as it quite probably will.

And this is a simple one-liner with a proper tool for the job, such as various yq tools, whose usage will be roughly (since, alas, they all differ): 

while read -r i
do
   yq < puppet-users.yaml "$i".password="'\!\!'" > puppet-users{new}.yaml
   mv puppet-users{new}.yaml puppet-users.yaml
done < user_list.txt

Further reading

Related Question