Sed/awk to remove double quotes ” that are inside of curly braces {}

awkcsvdeletesed

I have a csv file with data similar to this:

"1b579a5e-9701-40eb-bd36-2bc65169da99","week14_Friday-019","6907eaad-1aff-4d26-9088-ba20374b67c0","2181-019","f20af5bb-c716-42e0-9b9d-cbolf5bfecea","15-BIO-2001","COLLEGE Bio 1","d39330be-df56-4365-8fb4-37e68d040c52","Which engine has the smaller efficiency?","{choices:[a","b","c","d],"type:MultipleChoice}","{solution:[0],"selectAll:false,{"selectMultiple:false",}"type:MultipleChoice}","2016-04-25 00:30:19.000","1922ac5a-6ff6-4ea4-9078-6df4d85d294f","{solution:[0],"type:MultipleChoice}","1","1116911f-8ee5-45c3-b173-a6be681bb15a","FakeLastName","FakeFirstName","FakeName@mail.edu","Student"

I want to remove the double-quotes ", but only if they are inside of the curly braces {}, preferably with sed or awk. Desired output is:

"1b579a5e-9701-40eb-bd36-2bc65169da99","week14_Friday-019","6907eaad-1aff-4d26-9088-ba20374b67c0","2181-019","f20af5bb-c716-42e0-9b9d-cbolf5bfecea","15-BIO-2001","COLLEGE Bio 1","d39330be-df56-4365-8fb4-37e68d040c52","Which engine has the smaller efficiency?","{choices:[a,b,c,d],type:MultipleChoice}","{solution:[0],selectAll:false,{selectMultiple:false,}type:MultipleChoice}","2016-04-25 00:30:19.000","1922ac5a-6ff6-4ea4-9078-6df4d85d294f","{solution:[0],type:MultipleChoice}","1","1116911f-8ee5-45c3-b173-a6be681bb15a","FakeLastName","FakeFirstName","FakeName@mail.edu","Student"

Any help would be greatly appreciated. Thanks!

Best Answer

It would be easier with sed:

sed -e :1 -e 's/\({[^}]*\)"\([^}]*}\)/\1\2/g; t1'

Or perl:

perl -pe 's{\{.*?\}}{$& =~ s/"//gr}ge'

Note that it assumes there's no nested {...}.

To handle nested {...}, you can use perl's recursive regexp capabilities:

perl -pe 's(\{(?:[^{}]++|(?0))*\})($& =~ s/"//gr)ge'

With sed, working our way outwards to escape the inner {...}s before removing the "s:

sed 's/_/_u/g
     :1
     s/\({[^{}]*\){\([^{}]*\)}/\1_<\2_>/g; t1
     :2
     s/\({[^}]*\)"\([^}]*}\)/\1\2/g; t2
     s/_</{/g; s/_>/}/g;s/_u/_/g'

Using awk's `substr` function

This removes the first and last characters from each string:

$ systool -c fc_host -v | awk '/Class Device =/{host=substr($4,2,length($4)-2)}/port_state/{print host,substr($3,2,length($3)-2)}'
host1 Online

How it works:

In the code that you started with, there was the line

host=$4

In the revised code, that is replaced with:

host=substr($4,2,length($4)-2)

The substr function returns a substring of $4. In this case, it starts from the second character and extends a length of length($4)-2. Thus, this includes all characters except the first and last (which are the double quotes).

For the same reason, this command:

print host,$3)

was replaced with:

print host,substr($3,2,length($3)-2)

Using GNU awk's `gsub` function

Alternatively, gsub can be used to remove the double quotes:

$ systool -c fc_host -v | awk '/Class Device =/{gsub("\"","",$4);host=$4}/port_state/{gsub("\"","",$3);print host,$3}'
host1 Online

How it works

This is again just like the code that you started with but with two new commands added:

gsub("\"","",$4)
gsub("\"","",$3)

gsub does substitutions. In this case, it replaces " with an empty string, in effect removing the double quotes. On the first line above, it removes them from $4 (which is the host) and on the second line above, it removes them from $3 (which is the port_state).

Using awk's Field Separator

$ systool -c fc_host -v | awk -F'"' '/Class Device =/{host=$2} /port_state/{print host,$2}'
host1 Online

Using literal empty curly braces {} inside sed command from find -exec

In this case, you can work around find's exec grammar by capturing a brace expression and using a back reference in the replacement text:

$ cat f1 f2
f1: hello{}a
f2: hello{}a
$ find . -type f -exec sed -i 's/hello\([{][}]\)a/hello\1b/g' '{}' +
$ cat f1 f2
f1: hello{}b
f2: hello{}b

Or, more simply (as noted in the comments):

find "$dir" -type f -exec sed -i 's/\(hello[{]}\)a/\1b/g' {} +

Note that the -i option for Sed is not portable and will not work everywhere. The given command will work on GNU Sed only.

For details, see:

How to achieve portability with sed -i (in-place editing)?

Best Answer

Related Solutions

Remove Quotes (double or single) with Awk

Using awk's substr function