To all the 'sed' doctors out there:
How can you get 'sed' to extract a regular expression it has matched in a
line?
In other words words, I want just the string corresponding to the regular
expression with all the non-matching characters from the containing line stripped away.
I tried using the back-reference feature like below
regular expression to be isolated
gets `inserted`
here
|
v
sed -n 's/.*\( \).*/\1/p
this works for some expressions like
sed -n 's/.*\(CONFIG_[a-zA-Z0-9_]*\).*/\1/p
which neatly extracts all macro names starting with 'CONFIG_ ….' ( found in some '*.h' file ) and prints them all out line by line
CONFIG_AT91_GPIO
CONFIG_DRIVER_AT91EMAC
.
.
CONFIG_USB_ATMEL
CONFIG_USB_OHCI_NEW
.
e.t.c.
BUT the above breaks down for something like
sed -n 's/.*\([0-9][0-9]*\).*/\1/p
this always returns single digits like
7
9
.
.
6
rather than extracting a contiguous number field such as.
8908078
89670890
.
.
.
23019
.
e.t.c.
P.S.: I would be grateful to feedback on how this is achieved in 'sed'.
I know how to do this with 'grep' and 'awk'
I would like to find out if my – albeit limited – understanding of
'sed' has holes in it and if there is way to do this in 'sed' which I
have simply overlooked.
Best Answer
When a regexp contains groups, there may be more than one way to match a string against it: regexps with groups are ambiguous. For example, consider the regexp
^.*\([0-9][0-9]*\)$
and the stringa12
. There are two possibilities:a
against.*
and2
against[0-9]*
;1
is matched by[0-9]
.a1
against.*
and the empty string against[0-9]*
;2
is matched by[0-9]
.Sed, like all other regexp tools out there, applies the earliest longest match rule: it first tries to match the first variable-length portion against a string that's as long as possible. If it finds a way to match the rest of the string against the rest of the regexp, fine. Otherwise, sed tries the next longest match for the first variable-length portion and tries again.
Here, the match with the longest string first is
a1
against.*
, so the group only matches2
. If you want the group to start earlier, some regexp engines let you make the.*
less greedy, but sed doesn't have such a feature. So you need to remove the ambiguity with some additional anchor. Specify that the leading.*
cannot end with a digit, so that the first digit of the group is the first possible match.If the group of digits cannot be at the beginning of the line:
If the group of digits can be at the beginning of the line, and your sed supports the
\?
operator for optional parts:If the group of digits can be at the beginning of the line, sticking to standard regexp constructs:
By the way, it's that same earliest longest match rule that makes
[0-9]*
match the digits after the first one, rather than the subsequent.*
.Note that if there are multiple sequences of digits on a line, your program will always extract the last sequence of digits, again because of the earliest longest match rule applied to the initial
.*
. If you want to extract the first sequence of digits, you need to specify that what comes before is a sequence of non-digits.More generally, to extract the first match of a regexp, you need to compute the negation of that regexp. While this is always theoretically possible, the size of the negation grows exponentially with the size of the regexp you're negating, so this is often impractical.
Consider your other example:
This example actually exhibits the same issue, but you don't see it on typical inputs. If you feed it
hello CONFIG_FOO_CONFIG_BAR
, then the command above prints outCONFIG_BAR
, notCONFIG_FOO_CONFIG_BAR
.There's a way to print the first match with sed, but it's a little tricky:
(Assuming your sed supports
\n
to mean a newline in thes
replacement text.) This works because sed looks for the earliest match of the regexp, and we don't try to match what precedes theCONFIG_…
bit. Since there is no newline inside the line, we can use it as a temporary marker. TheT
command says to give up if the precedings
command didn't match.When you can't figure out how to do something in sed, turn to awk. The following command prints the earliest longest match of a regexp:
And if you feel like keeping it simple, use Perl.