Regular Expressions in Python 3

This chapter is from the book 

A regular expression is a compact notation for representing a collection of strings. What makes regular expressions so powerful is that a single regular expression can represent an unlimited number of strings—providing they meet the regular expression's requirements. Regular expressions (which we will mostly call "regexes" from now on) are defined using a mini-language that is completely different from Python—but Python includes the re module through which we can seamlessly create and use regexes.^*

Regexes are used for five main purposes:

• Parsing: identifying and extracting pieces of text that match certain criteria—regexes are used for creating ad hoc parsers and also by traditional parsing tools
• Searching: locating substrings that can have more than one form, for example, finding any of "pet.png", "pet.jpg", "pet.jpeg", or "pet.svg" while avoiding "carpet.png" and similar
• Searching and replacing: replacing everywhere the regex matches with a string, for example, finding "bicycle" or "human powered vehicle" and replacing either with "bike"
• Splitting strings: splitting a string at each place the regex matches, for example, splitting everywhere colon-space or equals (": " or "=") occurs
• Validation: checking whether a piece of text meets some criteria, for example, contains a currency symbol followed by digits

The regexes used for searching, splitting, and validation are often fairly small and understandable, making them ideal for these purposes. However, although regexes are widely and successfully used to create parsers, they do have a limitation in that area: They are only able to deal with recursively structured text if the maximum level of recursion is known. Also, large and complex regexes can be difficult to read and maintain. So apart from simple cases, for parsing the best approach is to use a tool designed for the purpose—for example, use a dedicated XML parser for XML. If such a parser isn't available, then an alternative to using regexes is to use a generic parsing tool, an approach that is covered in Chapter 14.

At its simplest a regular expression is an expression (e.g., a literal character), optionally followed by a quantifier. More complex regexes consist of any number of quantified expressions and may include assertions and may be influenced by flags.

This chapter's first section introduces and explains all the key regular expression concepts and shows pure regular expression syntax—it makes minimal reference to Python itself. Then the second section shows how to use regular expressions in the context of Python programming, drawing on all the material covered in the earlier sections. Readers familiar with regular expressions who just want to learn how they work in Python could skip to the second section ( 499). The chapter covers the complete regex language offered by the re module, including all the assertions and flags. We indicate regular expressions in the text using bold, show where they match using underlining, and show captures using shading.

Python's Regular Expression Language

In this section we look at the regular expression language in four subsections. The first subsection shows how to match individual characters or groups of characters, for example, match a, or match b, or match either a or b. The second subsection shows how to quantify matches, for example, match once, or match at least once, or match as many times as possible. The third subsection shows how to group subexpressions and how to capture matching text, and the final subsection shows how to use the language's assertions and flags to affect how regular expressions work.

Characters and Character Classes

The simplest expressions are just literal characters, such as a or 5, and if no quantifier is explicitly given it is taken to be "match one occurrence". For example, the regex tune consists of four expressions, each implicitly quantified to match once, so it matches one t followed by one u followed by one n followed by one e, and hence matches the strings tune and attuned.

Although most characters can be used as literals, some are "special characters"—these are symbols in the regex language and so must be escaped by preceding them with a backslash (\) to use them as literals. The special characters are \.^$?+*{}[]()|. Most of Python's standard string escapes can also be used within regexes, for example, \n for newline and \t for tab, as well as hexadecimal escapes for characters using the \xHH, \uHHHH, and \UHHHHHHHH syntaxes.

In many cases, rather than matching one particular character we want to match any one of a set of characters. This can be achieved by using a character class—one or more characters enclosed in square brackets. (This has nothing to do with a Python class, and is simply the regex term for "set of characters".) A character class is an expression, and like any other expression, if not explicitly quantified it matches exactly one character (which can be any of the characters in the character class). For example, the regex r[ea]d matches both red and radar, but not read. Similarly, to match a single digit we can use the regex [0123456789]. For convenience we can specify a range of characters using a hyphen, so the regex [0-9] also matches a digit. It is possible to negate the meaning of a character class by following the opening bracket with a caret, so [^0-9] matches any character that is not a digit.

Note that inside a character class, apart from \, the special characters lose their special meaning, although in the case of ^ it acquires a new meaning (negation) if it is the first character in the character class, and otherwise is simply a literal caret. Also, - signifies a character range unless it is the first character, in which case it is a literal hyphen.

Since some sets of characters are required so frequently, several have shorthand forms—these are shown in Table 13.1. With one exception the shorthands can be used inside character sets, so for example, the regex [\dA-Fa-f] matches any hexadecimal digit. The exception is . which is a shorthand outside a character class but matches a literal . inside a character class.

Table 13.1. Character Class Shorthands

Symbol	Meaning
.	Matches any character except newline; or any character at all with the re.DOTALL flag; or inside a character class matches a literal .
\d	Matches a Unicode digit; or [0-9] with the re.ASCII flag
\D	Matches a Unicode nondigit; or [^0-9] with the re.ASCII flag
\s	Matches a Unicode whitespace; or [ \t\n\r\f\v] with the re.ASCII flag
\S	Matches a Unicode nonwhitespace; or [^ \t\n\r\f\v] with the re.ASCII flag
\w	Matches a Unicode "word" character; or [a-zA-Z0-9_] with the re.ASCII flag
\W	Matches a Unicode non-"word" character; or [^a-zA-Z0-9_] with the re.ASCII flag

Quantifiers

A quantifier has the form {m,n} where m and n are the minimum and maximum times the expression the quantifier applies to must match. For example, both e{1,1}e{1,1} and e{2,2} match feel, but neither matches felt.

Writing a quantifier after every expression would soon become tedious, and is certainly difficult to read. Fortunately, the regex language supports several convenient shorthands. If only one number is given in the quantifier it is taken to be both the minimum and the maximum, so e{2} is the same as e{2,2}. And as we noted in the preceding section, if no quantifier is explicitly given, it is assumed to be one (i.e., {1,1} or {1}); therefore, ee is the same as e{1,1}e{1,1} and e{1}e{1}, so both e{2} and ee match feel but not felt.

Having a different minimum and maximum is often convenient. For example, to match travelled and traveled (both legitimate spellings), we could use either travel{1,2}ed or travell{0,1}ed. The {0,1} quantification is so often used that it has its own shorthand form, ?, so another way of writing the regex (and the one most likely to be used in practice) is travell?ed.

Two other quantification shorthands are provided: + which stands for {1,n} ("at least one") and * which stands for {0,n} ("any number of"); in both cases n is the maximum possible number allowed for a quantifier, usually at least 32767. All the quantifiers are shown in Table 13.2.

Table 13.2. Regular Expression Quantifiers

Syntax	Meaning
e? or e{0,1}	Greedily match zero or one occurrence of expression e
e?? or e{0,1}?	Nongreedily match zero or one occurrence of expression e
e+ or e{1,}	Greedily match one or more occurrences of expression e
e+? or e{1,}?	Nongreedily match one or more occurrences of expression e
e* or e{0,}	Greedily match zero or more occurrences of expression e
e*? or e{0,}?	Nongreedily match zero or more occurrences of expression e
e{m}	Match exactly m occurrences of expression e
e{m,}	Greedily match at least m occurrences of expression e
e{m,}?	Nongreedily match at least m occurrences of expression e
e{,n}	Greedily match at most n occurrences of expression e
e{,n}?	Nongreedily match at most n occurrences of expression e
e{m,n}	Greedily match at least m and at most n occurrences of expression e
e{m,n}?	Nongreedily match at least m and at most n occurrences of expression e

The + quantifier is very useful. For example, to match integers we could use \d+ since this matches one or more digits. This regex could match in two places in the string 4588.91, for example, 4588.91 and 4588.91. Sometimes typos are the result of pressing a key too long. We could use the regex bevel+ed to match the legitimate beveled and bevelled, and the incorrect bevellled. If we wanted to standardize on the one l spelling, and match only occurrences that had two or more ls, we could use bevell+ed to find them.

The * quantifier is less useful, simply because it can so often lead to unexpected results. For example, supposing that we want to find lines that contain comments in Python files, we might try searching for #*. But this regex will match any line whatsoever, including blank lines because the meaning is "match any number of #s"—and that includes none. As a rule of thumb for those new to regexes, avoid using * at all, and if you do use it (or if you use ?), make sure there is at least one other expression in the regex that has a nonzero quantifier—so at least one quantifier other than * or ? since both of these can match their expression zero times.

It is often possible to convert * uses to + uses and vice versa. For example, we could match "tasselled" with at least one l using tassell*ed or tassel+ed, and match those with two or more ls using tasselll*ed or tassell+ed.

If we use the regex \d+ it will match 136. But why does it match all the digits, rather than just the first one? By default, all quantifiers are greedy—they match as many characters as they can. We can make any quantifier nongreedy (also called minimal) by following it with a ? symbol. (The question mark has two different meanings—on its own it is a shorthand for the {0,1} quantifier, and when it follows a quantifier it tells the quantifier to be nongreedy.) For example, \d+? can match the string 136 in three different places: 136, 136, and 136. Here is another example: \d?? matches zero or one digits, but prefers to match none since it is nongreedy—on its own it suffers the same problem as * in that it will match nothing, that is, any text at all.

Nongreedy quantifiers can be useful for quick and dirty XML and HTML parsing. For example, to match all the image tags, writing <img.*> (match one "<", then one "i", then one "m", then one "g", then zero or more of any character apart from newline, then one ">") will not work because the .* part is greedy and will match everything including the tag's closing >, and will keep going until it reaches the last > in the entire text.

Three solutions present themselves (apart from using a proper parser). One is <img[^>]*> (match <img, then any number of non-> characters and then the tag's closing > character), another is <img.*?> (match <img, then any number of characters, but nongreedily, so it will stop immediately before the tag's closing >, and then the >), and a third combines both, as in <img[^>]*?>. None of them is correct, though, since they can all match <img>, which is not valid. Since we know that an image tag must have a src attribute, a more accurate regex is <img\s+[^>]*?src=\w+[^>]*?>. This matches the literal characters <img, then one or more whitespace characters, then nongreedily zero or more of anything except > (to skip any other attributes such as alt), then the src attribute (the literal characters src= then at least one "word" character), and then any other non-> characters (including none) to account for any other attributes, and finally the closing >.

Grouping and Capturing

In practical applications we often need regexes that can match any one of two or more alternatives, and we often need to capture the match or some part of the match for further processing. Also, we sometimes want a quantifier to apply to several expressions. All of these can be achieved by grouping with (), and in the case of alternatives using alternation with |.

Alternation is especially useful when we want to match any one of several quite different alternatives. For example, the regex aircraft|airplane|jet will match any text that contains "aircraft" or "airplane" or "jet". The same thing can be achieved using the regex air(craft|plane)|jet. Here, the parentheses are used to group expressions, so we have two outer expressions, air(craft|plane) and jet. The first of these has an inner expression, craft|plane, and because this is preceded by air the first outer expression can match only "aircraft" or "airplane".

Parentheses serve two different purposes—to group expressions and to capture the text that matches an expression. We will use the term group to refer to a grouped expression whether it captures or not, and capture and capture group to refer to a captured group. If we used the regex (aircraft|airplane|jet) it would not only match any of the three expressions, but would also capture whichever one was matched for later reference. Compare this with the regex (air(craft|plane)|jet) which has two captures if the first expression matches ("aircraft" or "airplane" as the first capture and "craft" or "plane" as the second capture), and one capture if the second expression matches ("jet"). We can switch off the capturing effect by following an opening parenthesis with ?:, so for example, (air(?:craft|plane)|jet) will have only one capture if it matches ("aircraft" or "airplane" or "jet").

A grouped expression is an expression and so can be quantified. Like any other expression the quantity is assumed to be one unless explicitly given. For example, if we have read a text file with lines of the form key=value, where each key is alphanumeric, the regex (\w+)=(.+) will match every line that has a nonempty key and a nonempty value. (Recall that . matches anything except newlines.) And for every line that matches, two captures are made, the first being the key and the second being the value.

For example, the key=value regular expression will match the entire line topic=physical geography with the two captures shown shaded. Notice that the second capture includes some whitespace, and that whitespace before the = is not accepted. We could refine the regex to be more flexible in accepting whitespace, and to strip off unwanted whitespace using a somewhat longer version:

   [ \t]*(\w+)[ \t]*=[ \t]*(.+)

This matches the same line as before and also lines that have whitespace around the = sign, but with the first capture having no leading or trailing whitespace, and the second capture having no leading whitespace. For example: topic= physical geography. We have been careful to keep the whitespace matching parts outside the capturing parentheses, and to allow for lines that have no whitespace at all. We did not use \s to match whitespace because that matches newlines (\n) which could lead to incorrect matches that span lines (e.g., if the re.MULTILINE flag is used). And for the value we did not use \S to match nonwhitespace because we want to allow for values that contain whitespace (e.g., English sentences). To avoid the second capture having trailing whitespace we would need a more sophisticated regex; we will see this in the next subsection.

Captures can be referred to using backreferences, that is, by referring back to an earlier capture group.^* One syntax for backreferences inside regexes themselves is \i where i is the capture number. Captures are numbered starting from one and increasing by one going from left to right as each new (capturing) left parenthesis is encountered. For example, to simplistically match duplicated words we can use the regex (\w+)\s+\1 which matches a "word", then at least one whitespace, and then the same word as was captured. (Capture number 0 is created automatically without the need for parentheses; it holds the entire match, that is, what we show underlined.) We will see a more sophisticated way to match duplicate words later.

In long or complicated regexes it is often more convenient to use names rather than numbers for captures. This can also make maintenance easier since adding or removing capturing parentheses may change the numbers but won't affect names. To name a capture we follow the opening parenthesis with ?P<name>. For example, (?P<key>\w+)=(?P<value>.+) has two captures called "key" and "value". The syntax for backreferences to named captures inside a regex is (?P=name). For example, (?P<word>\w+)\s+(?P=word) matches duplicate words using a capture called "word".

Assertions and Flags

One problem that affects many of the regexes we have looked at so far is that they can match more or different text than we intended. For example, the regex aircraft|airplane|jet will match "waterjet" and "jetski" as well as "jet". This kind of problem can be solved by using assertions. An assertion does not match any text, but instead says something about the text at the point where the assertion occurs.

One assertion is \b (word boundary), which asserts that the character that precedes it must be a "word" (\w) and the character that follows it must be a non"word" (\W), or vice versa. For example, although the regex jet can match twice in the text the jet and jetski are noisy, that is, the jet and jetski are noisy, the regex \bjet\b will match only once, the jet and jetski are noisy. In the context of the original regex, we could write it either as \baircraft\b|\bairplane\b|\bjet\b or more clearly as \b(?:aircraft|airplane|jet)\b, that is, word boundary, noncapturing expression, word boundary.

Many other assertions are supported, as shown in Table 13.3. We could use assertions to improve the clarity of a key=value regex, for example, by changing it to ^(\w+)=([^\n]+) and setting the re.MULTILINE flag to ensure that each key=value is taken from a single line with no possibility of spanning lines—providing no part of the regex matches a newline, so we can't use, say, \s. (The flags are shown in Table 13.5; 502; their syntaxes are described at the end of this subsection, and examples are given in the next section.) And if we want to strip whitespace from the ends and use named captures, the regex becomes:

   ^[ \t]*(?P<key>\w+)[ \t]*=[ \t]*(?P<value>[^\n]+)(?<![ \t])

Table 13.3. Regular Expression Assertions

Symbol	Meaning
^	Matches at the start; also matches after each newline with the re.MULTILINE flag
$	Matches at the end; also matches before each newline with the re.MULTILINE flag
\A	Matches at the start
\b	Matches at a "word" boundary; influenced by the re.ASCII flag—inside a character class this is the escape for the backspace character
\B	Matches at a non-"word" boundary; influenced by the re.ASCII flag
\Z	Matches at the end
(?=e)	Matches if the expression e matches at this assertion but does not advance over it—called lookahead or positive lookahead
(?!e)	Matches if the expression e does not match at this assertion and does not advance over it—called negative lookahead
(?<=e)	Matches if the expression e matches immediately before this assertion—called positive lookbehind
(?<!e)	Matches if the expression e does not match immediately before this assertion—called negative lookbehind

Even though this regex is designed for a fairly simple task, it looks quite complicated. One way to make it more maintainable is to include comments in it. This can be done by adding inline comments using the syntax (?#the comment), but in practice comments like this can easily make the regex even more difficult to read. A much nicer solution is to use the re.VERBOSE flag—this allows us to freely use whitespace and normal Python comments in regexes, with the one constraint that if we need to match whitespace we must either use \s or a character class such as []. Here's the key=value regex with comments:

   ^[ \t]*             # start of line and optional leading whitespace

   (?P<key>\w+)        # the key text

   [ \t]*=[ \t]*       # the equals with optional surrounding whitespace

   (?P<value>[^\n]+)   # the value text

   (?<![ \t])          # negative lookbehind to avoid trailing whitespace

In the context of a Python program we would normally write a regex like this inside a raw triple quoted string—raw so that we don't have to double up the backslashes, and triple quoted so that we can spread it over multiple lines.

In addition to the assertions we have discussed so far, there are additional assertions which look at the text in front of (or behind) the assertion to see whether it matches (or does not match) an expression we specify. The expressions that can be used in lookbehind assertions must be of fixed length (so the quantifiers ?, +, and * cannot be used, and numeric quantifiers must be of a fixed size, for example, {3}).

In the case of the key=value regex, the negative lookbehind assertion means that at the point it occurs the preceding character must not be a space or a tab. This has the effect of ensuring that the last character captured into the "value" capture group is not a space or tab (yet without preventing spaces or tabs from appearing inside the captured text).

Let's consider another example. Suppose we are reading a multiline text that contains the names "Helen Patricia Sharman", "Jim Sharman", "Sharman Joshi", "Helen Kelly", and so on, and we want to match "Helen Patricia", but only when referring to "Helen Patricia Sharman". The easiest way is to use the regex \b(Helen\s+Patricia)\s+Sharman\b. But we could also achieve the same thing using a lookahead assertion, for example, \b(Helen\s+Patricia)(?=\s+Sharman\b). This will match "Helen Patricia" only if it is preceded by a word boundary and followed by whitespace and "Sharman" ending at a word boundary.

To capture the particular variation of the forenames that is used ("Helen", "Helen P.", or "Helen Patricia"), we could make the regex slightly more sophisticated, for example, \b(Helen(?:\s+(?:P\.|Patricia))?)\s+(?=Sharman\b). This matches a word boundary followed by one of the forename forms—but only if this is followed by some whitespace and then "Sharman" and a word boundary.

Note that only two syntaxes perform capturing, (e) and (?P<name>e). None of the other parenthesized forms captures. This makes perfect sense for the lookahead and lookbehind assertions since they only make a statement about what follows or precedes them—they are not part of the match, but rather affect whether a match is made. It also makes sense for the last two parenthesized forms that we will now consider.

We saw earlier how we can backreference a capture inside a regex either by number (e.g., \1) or by name (e.g., (?P=name)). It is also possible to match conditionally depending on whether an earlier match occurred. The syntaxes are (?(id)yes_exp) and (?(id)yes_exp|no_exp). The id is the name or number of an earlier capture that we are referring to. If the capture succeeded the yes_exp will be matched here. If the capture failed the no_exp will be matched if it is given.

Let's consider an example. Suppose we want to extract the filenames referred to by the src attribute in HTML img tags. We will begin just by trying to match the src attribute, but unlike our earlier attempt we will account for the three forms that the attribute's value can take: single quoted, double quoted, and unquoted. Here is an initial attempt: src=(["'])([^"'>]+)\1. The ([^"'>]+) part captures a greedy match of at least one character that isn't a quote or >. This regex works fine for quoted filenames, and thanks to the \1 matches only when the opening and closing quotes are the same. But it does not allow for unquoted filenames. To fix this we must make the opening quote optional and therefore match only if it is present.

Here is a revised regex: src=(["'])?([^"'>]+)(?(1)\1). We did not provide a no_exp since there is nothing to match if no quote is given. Unfortunately, this doesn't work quite right. It will work fine for quoted filenames, but for unquoted filenames it will work only if the src attribute is the last attribute in the tag; otherwise it will incorrectly match text into the next attribute. The solution is to treat the two cases (quoted and unquoted) separately, and to use alternation: src=((["'])([^\1>]+?)\1|([^"' >]+)). Now let's see the regex in context, complete with named groups, nonmatching parentheses, and comments:

<img\s+                    # start of the tag
[^>]*?                     # any attributes that precede the src
src=                       # start of the src attribute
(?:
    (?P<quote>["'])        # opening quote
    (?P<qimage>[^\1>]+?)   # image filename
    (?P=quote)             # closing quote matching the opening quote
    |                          # ---or alternatively---
    (?P<uimage>[^"' >]+)   # unquoted image filename
)
[^>]*?                     # any attributes that follow the src
>                          # end of the tag

The indentation is just for clarity. The noncapturing parentheses are used for alternation. The first alternative matches a quote (either single or double), then the image filename (which may contain any characters except for the quote that matched or >), and finally, another quote which must be the same as the matching quote. We also had to use minimal matching, +?, for the file-name, to ensure that the match doesn't extend beyond the first matching closing quote. This means that a filename such as "I'm here!.png" will match correctly. Note also that to refer to the matching quote inside the character class we had to use a numbered backreference, \1, instead of (?P=quote), since only numbered backreferences work inside character classes. The second alternative matches an unquoted filename—a string of characters that don't include quotes, spaces, or >. Due to the alternation, the filename is captured in "qimage" (capture number 2) or in "uimage" (capture number 3, since (?P=quote) matches but doesn't capture), so we must check for both.

The final piece of regex syntax that Python's regular expression engine offers is a means of setting the flags. Usually the flags are set by passing them as additional parameters when calling the re.compile() function, but sometimes it is more convenient to set them as part of the regex itself. The syntax is simply (?flags) where flags is one or more of a (the same as passing re.ASCII), i (re.IGNORECASE), m (re.MULTILINE), s (re.DOTALL), and x (re.VERBOSE).^* If the flags are set this way they should be put at the start of the regex; they match nothing, so their effect on the regex is only to set the flags.