Why can’t a constant regular expression be put on the left side of an ~ operator in gawk

gawk

Why can't I put a regular expression on the left side of the ~ operator when using gawk?

For example, given the file below with fields delimited with tabs(\t):

$ cat cats
siberian    1970    73  2500
shorthair   1999    60  3000
longhair    1998    102 9859
scottish    2001    30  6000

If I use gawk to find a record, it works:

$ gawk '$1 ~ /h/' cats
shorthair   1999    60  3000
longhair    1998    102 9859
scottish    2001    30  6000

However if I move the operands $1 and /h/ around, it doesn't:

$ gawk '/h/ ~ $1' cats
gawk: cmd. line:1: warning: regular expression on left of `~' or `!~' operator

The gawk man page for the ~ operator says:

Regular expression match, negated match. NOTE: Do not use a constant
regular expression (/foo/) on the left-hand side of a ~ or !~. Only
use one on the right-hand side. The expression /foo/ ~ exp has the
same meaning as (($0 ~ /foo/) ~ exp). This is usually not what was
intended.

I don't understand how the expression /foo/ is evaluated to become ($0 ~ /foo/) and also this seems to only imply the weaker phrase "bad things will happen if you put a constant regular expression on the left" it doesn't actually imply the stronger phrase of "the behaviour of gawk is undefined if you put a constant regular expression on the left because it wasn't programmed to be used in this way".

I basically don't understand how the operator ~ is evaluated internally.

Best Answer

To quote the POSIX spec for awk:

When an ERE token appears as an expression in any context other than as the right-hand of the ~ or !~ operator or as one of the built-in function arguments described below, the value of the resulting expression shall be the equivalent of:

$0 ~ /ere/

This (combined with the action defaulting to { print }) is why you can use awk as a grep substitute by just doing awk '/b/' <file.

So, the answer is just "it's defined to work that way". /ere/ is defined to mean $0 ~ /ere/ except in certain circumstances, and /ere/ ~ $1 is not one of the exceptional circumstances, so it gets evaluated as ($0 ~ /ere/) ~ $1.

Examples

I simplified the regex to provide examples of the different behavior:

$ echo -e " a\n ab.conf\n <ac.conf" | gawk '/\<(a|x)/ { print}'
 a
 ab.conf
 <ac.conf
$ echo -e " a\n ab.conf\n <ac.conf" | mawk '/\<(a|x)/ { print}'
 <ac.conf

Again, gawk interprets \< as the beginning of a word while mawk interprets it simply as an angle bracket.

What does POSIX say about this issue

The GNU awk manual explains:

If you place a backslash in a string constant before something that is not one of the characters previously listed, POSIX awk purposely leaves what happens as undefined.

In other words, in this case, the different awk interpreters are free to make their own decisions.

Why does gawk treat `0123` as a decimal number when coming from the input data

This is related to the generalised strnum handling in version 4.2 of GAWK.

Input values which look like numbers are treated as strnum values, represented internally as having both string and number types. “0123” qualifies as looking like a number, so it is handled as a strnum. strtonum is designed to handle both string and number inputs; it looks for numbers first, and when it encounters an input number, returns the number without transformation:

NODE *
do_strtonum(int nargs)
{
        NODE *tmp;
        AWKNUM d;

        tmp = fixtype(POP_SCALAR());
        if ((tmp->flags & NUMBER) != 0)
                d = (AWKNUM) tmp->numbr;
        else if (get_numbase(tmp->stptr, tmp->stlen, use_lc_numeric) != 10)
                d = nondec2awknum(tmp->stptr, tmp->stlen, NULL);
        else
                d = (AWKNUM) force_number(tmp)->numbr;

        DEREF(tmp);
        return make_number((AWKNUM) d);
}

Thus “0123” becomes the number 123, and strtonum returns that directly.

“0x123” doesn’t look like a number (by the rules defined in the link given above), so it is handled as a string and processed as you’d expect by strtonum.

A number is defined as follows in AWK:

The input string is decomposed into two parts: an initial, possibly empty, sequence of white-space characters (as specified by isspace()) and a subject sequence interpreted as a floating-point constant.

The expected form of the subject sequence is an optional '+' or '-' sign, then a non-empty sequence of digits optionally containing a <period>, then an optional exponent part. An exponent part consists of 'e' or 'E', followed by an optional sign, followed by one or more decimal digits.

The sequence starting with the first digit or the <period> (whichever occurs first) is interpreted as a floating constant of the C language, and if neither an exponent part nor a <period> appears, a is assumed to follow the last digit in the string. If the subject sequence begins with a <hyphen-minus>, the value resulting from the conversion is negated.

Best Answer

Related Solutions

Problem with regular expression in gawk (‘\<' not working)

Examples

What does POSIX say about this issue

Why does gawk treat `0123` as a decimal number when coming from the input data

Related Question