Regular Expressions in Python 3

This chapter is from the book

Programming in Python 3: A Complete Introduction to the Python Language, 2nd Edition

Learn More Buy

This chapter is from the book

This chapter is from the book 

Programming in Python 3: A Complete Introduction to the Python Language, 2nd Edition

Learn More Buy

The Regular Expression Module

The re module provides two ways of working with regexes. One is to use the functions listed in Table 13.4 ( 502), where each function is given a regex as its first argument. Each function converts the regex into an internal format—a process called compiling—and then does its work. This is very convenient for one-off uses, but if we need to use the same regex repeatedly we can avoid the cost of compiling it at each use by compiling it once using the re.compile() function. We can then call methods on the compiled regex object as many times as we like. The compiled regex methods are listed in Table 13.6 ( 503).

Table 13.4. The Regular Expression Module's Functions

Table 13.5. The Regular Expression Module's Flags

Table 13.6. Regular Expression Object Methods

match = re.search(r"#[\dA-Fa-f]{6}\b", text)

This code snippet shows the use of an re module function. The regex matches HTML-style colors (such as #C0C0AB). If a match is found the re.search() function returns a match object; otherwise, it returns None. The methods provided by match objects are listed in Table 13.7 ( 507).

Table 13.7. Match Object Attributes and Methods

If we were going to use this regex repeatedly, we could compile it once and then use the compiled regex whenever we needed it:

color_re = re.compile(r"#[\dA-Fa-f]{6}\b")
match = color_re.search(text)

As we noted earlier, we use raw strings to avoid having to escape backslashes. Another way of writing this regex would be to use the character class [\dA-F] and pass the re.IGNORECASE flag as the last argument to the re.compile() call, or to use the regex (?i)#[\dA-F]{6}\b which starts with the ignore case flag.

If more than one flag is required they can be combined using the OR operator (|), for example, re.MULTILINE|re.DOTALL, or (?ms) if embedded in the regex itself.

We will round off this section by reviewing some examples, starting with some of the regexes shown in earlier sections, so as to illustrate the most commonly used functionality that the re module provides. Let's start with a regex to spot duplicate words:

double_word_re = re.compile(r"\b(?P<word>\w+)\s+(?P=word)(?!\w)",
                            re.IGNORECASE)
for match in double_word_re.finditer(text):
    print("{0} is duplicated".format(match.group("word")))

The regex is slightly more sophisticated than the version we made earlier. It starts at a word boundary (to ensure that each match starts at the beginning of a word), then greedily matches one or more "word" characters, then one or more whitespace characters, then the same word again—but only if the second occurrence of the word is not followed by a word character.

If the input text was "win in vain", without the first assertion there would be one match and one capture: win in vain. There aren't two matches because while (?P<word>) matches and captures, the \s+ and (?P=word) parts only match. The use of the word boundary assertion ensures that the first word matched is a whole word, so we end up with no match or capture since there is no duplicate whole word. Similarly, if the input text was "one and and two let's say", without the last assertion there would be two matches and two captures: one and and two let's say. The use of the lookahead assertion means that the second word matched is a whole word, so we end up with one match and one capture: one and and two let's say.

The for loop iterates over every match object returned by the finditer() method and we use the match object's group() method to retrieve the captured group's text. We could just as easily (but less maintainably) have used group(1)—in which case we need not have named the capture group at all and just used the regex \b(\w+)\s+\1(?!\w). Another point to note is that we could have used a word boundary \b at the end, instead of (?!\w).

Another example we presented earlier was a regex for finding the filenames in HTML image tags. Here is how we would compile the regex, adding flags so that it is not case-sensitive, and allowing us to include comments:

image_re = re.compile(r"""
            <img\s+                     # start of tag
            [^>]*?                      # non-src attributes
            src=                        # start of src attribute
            (?:
                (?P<quote>["'])         # opening quote
                (?P<qimage>[^\1>]+?)    # image filename
                (?P=quote)              # closing quote
            |                           # ---or alternatively---
                (?P<uimage>[^"' >]+)    # unquoted image filename
            )
            [^>]*?                      # non-src attributes
            >                           # end of the tag
            """, re.IGNORECASE|re.VERBOSE)
image_files = []
for match in image_re.finditer(text):
    image_files.append(match.group("qimage") or
                       match.group("uimage"))

Again we use the finditer() method to retrieve each match and the match object's group() function to retrieve the captured texts. Each time a match is made we don't know which of the image groups ("qimage" or "uimage") has matched, but using the or operator provides a neat solution for this. Since the case insensitivity applies only to img and src, we could drop the re.IGNORECASE flag and use [Ii][Mm][Gg] and [Ss][Rr][Cc] instead. Although this would make the regex less clear, it might make it faster since it would not require the text being matched to be set to upper- (or lower-) case—but it is likely to make a difference only if the regex was being used on a very large amount of text.

One common task is to take an HTML text and output just the plain text that it contains. Naturally we could do this using one of Python's parsers, but a simple tool can be created using regexes. There are three tasks that need to be done: delete any tags, replace entities with the characters they represent, and insert blank lines to separate paragraphs. Here is a function (taken from the html2text.py program) that does the job:

def html2text(html_text):
    def char_from_entity(match):
        code = html.entities.name2codepoint.get(match.group(1), 0xFFFD)
        return chr(code)
text = re.sub(r"<!--(?:.|\n)*?-->", "", html_text)          #1
text = re.sub(r"<[Pp][^>]*?>", "\n\n", text)                #2
text = re.sub(r"<[^>]*?>", "", text)                        #3
text = re.sub(r"&#(\d+);", lambda m: chr(int(m.group(1))), text)
text = re.sub(r"&([A-Za-z]+);", char_from_entity, text)      #5
text = re.sub(r"\n(?:[ \xA0\t]+\n)+;", "\n", text)           #6
return re.sub(r"\n\n+", "\n\n", text.strip())               #7

The first regex, <!--(?:.|\n)*?-->, matches HTML comments, including those with other HTML tags nested inside them. The re.sub() function replaces as many matches as it finds with the replacement—deleting the matches if the replacement is an empty string, as it is here. (We can specify a maximum number of matches by giving an additional integer argument at the end.)

We are careful to use nongreedy (minimal) matching to ensure that we delete one comment for each match; if we did not do this we would delete from the start of the first comment to the end of the last comment.

The second regex, <[Pp][^>]*?>, matches opening paragraph tags (such as <P> or <p align="center">). It matches the opening <p (or <P), then any attributes (using nongreedy matching), and finally the closing >. The second call to the re.sub() function uses this regex to replace opening paragraph tags with two newline characters (the standard way to delimit a paragraph in a plain text file).

The third regex, <[^>]*?>, matches any tag and is used in the third re.sub() call to delete all the remaining tags.

HTML entities are a way of specifying non-ASCII characters using ASCII characters. They come in two forms: &name; where name is the name of the character—for example, &copy; for ©—and &#digits; where digits are decimal digits identifying the Unicode code point—for example, &#165; for ¥. The fourth call to re.sub() uses the regex &#(\d+);, which matches the digits form and captures the digits into capture group 1. Instead of a literal replacement text we have passed a lambda function. When a function is passed to re.sub() it calls the function once for each time it matches, passing the match object as the function's sole argument. Inside the lambda function we retrieve the digits (as a string), convert to an integer using the built-in int() function, and then use the built-in chr() function to obtain the Unicode character for the given code point. The function's return value (or in the case of a lambda expression, the result of the expression) is used as the replacement text.

The fifth re.sub() call uses the regex &([A-Za-z]+); to capture named entities. The standard library's html.entities module contains dictionaries of entities, including name2codepoint whose keys are entity names and whose values are integer code points. The re.sub() function calls the local char_from_entity() function every time it has a match. The char_from_entity() function uses dict.get() with a default argument of 0xFFFD (the code point of the standard Unicode replacement character—often depicted as ). This ensures that a code point is always retrieved and it is used with the chr() function to return a suitable character to replace the named entity with—using the Unicode replacement character if the entity name is invalid.

The sixth re.sub() call's regex, \n(?:[ \xA0\t]+\n)+, is used to delete lines that contain only whitespace. The character class we have used contains a space, a nonbreaking space (which &nbsp; entities are replaced with in the preceding regex), and a tab. The regex matches a newline (the one at the end of a line that precedes one or more whitespace-only lines), then at least one (and as many as possible) lines that contain only whitespace. Since the match includes the newline, from the line preceding the whitespace-only lines we must replace the match with a single newline; otherwise, we would delete not just the whitespace-only lines but also the newline of the line that preceded them.

The result of the seventh and last re.sub() call is returned to the caller. This regex, \n\n+, is used to replace sequences of two or more newlines with exactly two newlines, that is, to ensure that each paragraph is separated by just one blank line.

In the HTML example none of the replacements were directly taken from the match (although HTML entity names and numbers were used), but in some situations the replacement might need to include all or some of the matching text. For example, if we have a list of names, each of the form Forename Middlename1 ... MiddlenameN Surname, where there may be any number of middle names (including none), and we want to produce a new version of the list with each item of the form Surname, ForenameMiddlename1...MiddlenameN, we can easily do so using a regex:

new_names = []
for name in names:
    name = re.sub(r"(\w+(?:\s+\w+)*)\s+(\w+)", r"\2, \1", name)
    new_names.append(name)

The first part of the regex, (\w+(?:\s+\w+)*), matches the forename with the first \w+ expression and zero or more middle names with the (?:\s+\w+)* expression. The middle name expression matches zero or more occurrences of whitespace followed by a word. The second part of the regex, \s+(\w+), matches the whitespace that follows the forename (and middle names) and the surname.

If the regex looks a bit too much like line noise, we can use named capture groups to improve legibility and make it more maintainable:

name = re.sub(r"(?P<forenames>\w+(?:\s+\w+)*)"
              r"\s+(?P<surname>\w+)",
              r"\g<surname>, \g<forenames>", name)

Captured text can be referred to in a sub() or subn() function or method by using the syntax \i or \g<id> where i is the number of the capture group and id is the name or number of the capture group—so \1 is the same as \g<1>, and in this example, the same as \g<forenames>. This syntax can also be used in the string passed to a match object's expand() method.

Why doesn't the first part of the regex grab the entire name? After all, it is using greedy matching. In fact it will, but then the match will fail because although the middle names part can match zero or more times, the surname part must match exactly once, but the greedy middle names part has grabbed everything. Having failed, the regular expression engine will then backtrack, giving up the last "middle name" and thus allowing the surname to match. Although greedy matches match as much as possible, they stop if matching more would make the match fail.

For example, if the name is "John le Carré", the regex will first match the entire name, that is, John le Carré. This satisfies the first part of the regex but leaves nothing for the surname part to match, and since the surname is mandatory (it has an implicit quantifier of 1), the regex has failed. Since the middle names part is quantified by *, it can match zero or more times (currently it is matching twice, " le" and " Carré"), so the regular expression engine can make it give up some of its match without causing it to fail. Therefore, the regex backtracks, giving up the last \s+\w+ (i.e., " Carré"), so the match becomes John le Carré with the match satisfying the whole regex and with the two match groups containing the correct texts.

There's one weakness in the regex as written: It doesn't cope correctly with forenames that are written using an initial, such as "James W. Loewen", or "J. R. R. Tolkein". This is because \w matches word characters and these don't include period. One obvious—but incorrect—solution is to change the forenames part of the regex's \w+ expression to [\w.]+, in both places that it occurs. A period in a character class is taken to be a literal period, and character class shorthands retain their meaning inside character classes, so the new expression matches word characters or periods. But this would allow for names like ".", "..", ".A", ".A.", and so on. In view of this, a more subtle approach is required.

name = re.sub(r"(?P<forenames>\w+\.?(?:\s+\w+\.?)*)"
              r"\s+(?P<surname>\w+)",
              r"\g<surname>, \g<forenames>", name)

Here we have changed the forenames part of the regex (the first line). The first part of the forenames regex matches one or more word characters optionally followed by a period. The second part matches at least one whitespace character, then one or more word characters optionally followed by a period, with the whole of this second part itself matching zero or more times.

When we use alternation (|) with two or more alternatives capturing, we don't know which alternative matched, so we don't know which capture group to retrieve the captured text from. We can of course iterate over all the groups to find the nonempty one, but quite often in this situation the match object's lastindex attribute can give us the number of the group we want. We will look at one last example to illustrate this and to give us a little bit more regex practice.

Suppose we want to find out what encoding an HTML, XML, or Python file is using. We could open the file in binary mode, and read, say, the first 1000 bytes into a bytes object. We could then close the file, look for an encoding in the bytes, and reopen the file in text mode using the encoding we found or using a fallback encoding (such as UTF-8). The regex engine expects regexes to be supplied as strings, but the text the regex is applied to can be a str, bytes, or bytearray object, and when bytes or bytearray objects are used, all the functions and methods return bytes instead of strings, and the re.ASCII flag is implicitly switched on.

For HTML files the encoding is normally specified in a <meta> tag (if specified at all), for example, <meta http-equiv='Content-Type' content='text/html; charset=ISO-8859-1'/>. XML files are UTF-8 by default, but this can be overridden, for example, <?xml version="1.0" encoding="Shift_JIS"?>. Python 3 files are also UTF-8 by default, but again this can be overridden by including a line such as # encoding: latin1 or # -*- coding: latin1 -*- immediately after the shebang line.

Here is how we would find the encoding, assuming that the variable binary is a bytes object containing the first 1000 bytes of an HTML, XML, or Python file:

match = re.search(r"""(?<![-\w])                    #1
                      (?:(?:en)?coding|charset)     #2
                      (?:=(["'])?([-\w]+)(?(1)\1)   #3
                      |:\s*([-\w]+))""".encode("utf8"),
                  binary, re.IGNORECASE|re.VERBOSE)
encoding = match.group(match.lastindex) if match else b"utf8"

To search a bytes object we must specify a pattern that is also a bytes object. In this case we want the convenience of using a raw string, so we use one and convert it to a bytes object as the re.search() function's first argument.

The first part of the regex itself is a lookbehind assertion that says that the match cannot be preceded by a hyphen or a word character. The second part matches "encoding", "coding", or "charset" and could have been written as (?:encoding|coding|charset). We have made the third part span two lines to emphasise the fact that it has two alternating parts, =(["'])?([-\w]+)(?(1)\1) and :\s*([-\w]+), only one of which can match. The first of these matches an equals sign followed by one or more word or hyphen characters (optionally enclosed in matching quotes using a conditional match), and the second matches a colon and then optional whitespace followed by one or more word or hyphen characters. (Recall that a hyphen inside a character class is taken to be a literal hyphen if it is the first character; otherwise, it means a range of characters, for example, [0-9].)

We have used the re.IGNORECASE flag to avoid having to write (?:(?:[Ee][Nn])? [Cc][Oo][Dd][Ii][Nn][Gg]|[Cc][Hh][Aa][Rr][Ss][Ee][Tt]) and we have used the re.VERBOSE flag so that we can lay out the regex neatly and include comments (in this case just numbers to make the parts easy to refer to in this text).

There are three capturing match groups, all in the third part: (["'])? which captures the optional opening quote, ([-\w]+) which captures an encoding that follows an equals sign, and the second ([-\w]+) (on the following line) that captures an encoding that follows a colon. We are only interested in the encoding, so we want to retrieve either the second or third capture group, only one of which can match since they are alternatives. The lastindex attribute holds the index of the last matching capture group (either 2 or 3 when a match occurs in this example), so we retrieve whichever matched, or use a default encoding if no match was made.

We have now seen all of the most frequently used re module functionality in action, so we will conclude this section by mentioning one last function. The re.split() function (or the regex object's split() method) can split strings based on a regex. One common requirement is to split a text on whitespace to get a list of words. This can be done using re.split(r"\s+", text) which returns a list of words (or more precisely a list of strings, each of which matches \S+). Regular expressions are very powerful and useful, and once they are learned, it is easy to see all text problems as requiring a regex solution. But sometimes using string methods is both sufficient and more appropriate. For example, we can just as easily split on whitespace by using text.split() since the str.split() method's default behavior (or with a first argument of None) is to split on \s+.