Image

Imageplanet_python

Ned Batchelder: Regex affordances

Python regexes have a number of features that bring new power to text manipulation. I’m not talking about fancy matching features like negative look-behinds, but ways you can construct and use regexes. As a demonstration, I’ll show you some real code from a real project.

Coverage.py will expand environment variables in values read from its configuration files. It does this with a function calledsubstitute_variables:

Call it with a string and a dictionary, and it makes the substitutions:

We use a regex to pick apart the text:

This isn’t a super-fancy regex: it doesn’t use advanced pattern matching. But there are some useful regex features at work here:

  • The(?x)flag at the beginning turns on “verbose” regex syntax. In this mode, all white space is ignored so the regex can be multi-line and we can indent to help see the structure, and comments are allowed at the ends of lines.
  • Named groups like(?P<word1> … )are used to capture parts of the text that we can retrieve later by name.
  • There are also two groups used to get the precedence of operators right, but we don’t want to capture those values separately, so I use the non-capturing group syntax for them:(?: … ). In this code, we only ever access groups by name, so I could have left them as regular capturing groups, but I think it’s clearer to indicate up-front that we won’t be using them.

The verbose syntax in particular makes it easier to understand the regex. Compare to what it would look like in one line:

Once we have the regex, we can usere.sub()to replace the variables with their values:

But we’re going to use another power feature of Python regexes:dollar_replacehere isn’t a string, it’s a function! Each fragment the regex matches will be passed as a match object to ourdollar_replacefunction. It returns a string which re.sub() uses as the replacement in the text:

First we usematch.group(). Called with a number of names, it returns a tuple of what those named groups matched. They could be the matched text, or None if the group didn’t match anything.

The way our regex is written only one of those three groups will match, so the tuple will have one string and two None’s. To get the matched string, we usenext()to find it. If the built-inany()returned the first true thing it found this code could be simpler, but it doesn’t so we have to do it this way.

Now we can check the value to decide on the replacement:

  • If the match was a dollar sign, we return a dollar sign.
  • If the word is one of our defined variables, we return the value of the variable.
  • Since the word isn’t a defined variable, we check if the “strict” marker was found, and if so, raise an exception.
  • Otherwise we return the default value provided.

The final piece of the implementation is to usere.sub()and return the result:

Regexes are often criticized for being too opaque and esoteric. But done right, they can be very powerful and don’t have to be a burden. What we’ve done here is used simple pattern matching paired with useful API features to compactly write a useful transformation.

BTW, if you are interested, thereal code is in coverage.py.

https://nedbatchelder.com/blog/202504/regex_affordances.html

LJ Video