Why do some regex commands have opposite intepretations of ‘\’ with various characters

findregular expression

Take, for example, this command:

find . -regex ".*\.\(cpp\|h\)"

This will find all the .h and .cpp files in your directory. The period character '.' in regular expressions usually means "any character". To get it to match only an actual period, you must escape it using the backslash character '\'.

In this case, given a character with a special meaning, you must escape it to get the actual character it represents.

Now, take the parenthesis and the "or" bar, being characters '(', ')', and '|', respectively. These also have special meanings, used for grouping regular expressions. However, to get the special meaning, the characters must be escaped using the backslash! Without the backslash, the characters have the meaning of the actual character it represents.

Why is the '.' treated differently from '(', ')', and '|'?

Best Answer

The answer is really "just because". There's a whole bunch of different regular expression syntaxes, and while they share a similar appearance and usually the basics are the same, they vary in the particulars.

Historically, every tool had its own new implementation, doing whatever the author thought best. There's a balance between making characters special with and without escaping — too many characters that are "naturally special" and you end up having to escape them all the time just to match on them; or, the other way around, you end up needing a bunch of escapes to use common regex syntax like () grouping. And everyone writing a program decided how to do it based on the needs of what their program matched against, on what they felt was the right approach, and on the phase of the moon.

There's an attempt at standardization from POSIX, which defines "basic regular expressions" and "extended regular expressions". Awesomely, these work backwards from each other in regards to \ — sometimes, but not with perfect consistency.

Perl regular expressions have become another defacto standard, for two reasons: first, they're very flexible and powerful, and second, they're actually pretty sane, with conventions like "\ always escapes a non-alphanumeric character".

GNU Find has a -regextype option, where you can change the regular expression syntax used. Sadly, "perl" is not an option, at least in the version of find I have. (The default is, not surprisingly from GNU, "emacs", and that syntax is documented here.)

Related Solutions

Shell – Number of backslashes needed for escaping regex backslash on the command-line

For the unquoted example, each \\ pair passes one backslash to grep, so 4 backslashes pass two to grep, which translates to a single backslash. 6 backslashes pass three to grep, translating to one backslash and one \c, which is equal to c. One additional backslash does not change anything, because it is translated \c -> c by the shell. Eight backslashes in the shell are four in grep, translated to two, so this does not match anymore.

For the example in double quotes, note what follows your second quote from the bash manpage:

The backslash retains its special meaning only when followed by one of the following characters: $, `, ", \, or newline.

I.e. when you give an odd number of backslashes, the sequence ends in \c, which would be equal to c in the unquoted case, but when quoted, the backslash looses its special meaning, so \c is passed to grep. That is why the range of "possible" backslashes (i.e. those that make up a pattern matching your example file) slides down by one.

Need to escape regex characters in sed to be interpreted as regex characters

This is because sed uses POSIX BREs (Basic Regular Expressions) as opposed to the EREs (Extended Regular Expressions) you're probably used to from Perl or friends.

From the sed(1) man page:

REGULAR EXPRESSIONS
       POSIX.2 BREs should be supported, but they aren't completely because of
       performance problems.  The \n sequence in a regular expression  matches
       the newline character, and similarly for \a, \t, and other sequences.

Relevant quote from the above link:

The Basic Regular Expressions or BRE flavor standardizes a flavor similar to the one used by the traditional UNIX grep command. This is pretty much the oldest regular expression flavor still in use today. One thing that sets this flavor apart is that most metacharacters require a backslash to give the metacharacter its flavor. Most other flavors, including POSIX ERE, use a backslash to suppress the meaning of metacharacters.

Quoted verbatim from Craig Sanders' comment:

Note that in GNU sed at least, you can tell sed to use extended regexps with the -r or --regexp-extended command line option. This is useful if you want to avoid uglifying your sed script with excessive escaping.

Best Answer

Related Solutions

Shell – Number of backslashes needed for escaping regex backslash on the command-line

Need to escape regex characters in sed to be interpreted as regex characters

Related Question