Regular Expressions in Python

This chapter is from the book

Summerfield:Programm in Python 3_p1

Learn More Buy

This chapter is from the book

This chapter is from the book 

Summerfield:Programm in Python 3_p1

Learn More Buy

The Regular Expression Module

The re module provides two ways of working with regexes. One is to use the functions listed in Table 5, in which each function is given a regex as its first argument. Each function converts the regex into an internal format—a process called compiling—and then does its work. This technique is very convenient for one-off uses, but if we need to use the same regex repeatedly we can avoid the cost of compiling it at each use by compiling it once using the re.compile() function. Then we can call methods on the compiled regex object as many times as we like. (Table 6 lists the compiled regex methods.)

Table 5 The Regular Expression Module's Functions

Table 6 Regular Expression Object Methods

The following code snippet shows the use of the re module function:

match = re.search(r"#[\dA-Fa-f]{6}\b", text)

The regex matches HTML-style colors (such as #C0C0AB). If a match is found, the re.search() function returns a match object; otherwise, it returns None. The methods provided by match objects are listed in Table 7.

If we were going to use this regex repeatedly, we could compile it once and then use the compiled regex whenever we needed it:

color_re = re.compile(r"#[\dA-Fa-f]{6}\b")
match = color_re.search(text)

As noted earlier, we use raw strings to avoid having to escape backslashes. Another way of writing this regex would be to use the character class [\dA-F] and pass the re.IGNORECASE flag as the last argument to the re.compile() call, or to use this regex:

(?i)#[\dA-F]{6}\b

which starts with the ignore case flag.

If more than one flag is required, they can be combined using the OR operator (|); for example, re.MULTILINE|re.DOTALL, or (?ms) if embedded in the regex itself.

Table 7 Match Object Attributes and Methods

We'll round off this section by reviewing some examples, starting with some of the regexes shown in earlier sections, so as to illustrate the most commonly used functionality that the re module provides. Let's start with a regex to spot duplicate words:

double_word_re = re.compile(r"\b(?P<word>\w+)\s+(?P=word)(?!\w)",
                            re.IGNORECASE)
for match in double_word_re.finditer(text):
    print("{0} is duplicated".format(match.group("word")))

This regex is slightly more sophisticated than the version we made earlier. It starts at a word boundary (to ensure that each match starts at the beginning of a word), and then greedily matches one or more "word" characters, one or more whitespace characters, and then the same word again—but only if the second occurrence of the word is not followed by a word character.

If the input text was win in vain, without the first assertion there would be one match and two captures: win in vain. The use of the word boundary assertion ensures that the first word matched is a whole word, so we end up with no match or capture, since there's no duplicate word. Similarly, if the input text was one and and two let's say, without the last assertion there would be two matches and two captures: one and and two let's say. The use of the lookahead assertion means that the second word matched is a whole word, so we end up with one match and one capture: one and and two let's say.

The for loop iterates over every match object returned by the finditer() method, and we use the match object's group() method to retrieve the captured group's text. We could just as easily (but less maintainably) have used group(1)—in which case we need not have named the capture group at all and just used the following regex:

(\w+)\s+\1(?!\w)

Another point to note is that we could have used a word boundary \b at the end, instead of (?!\w).

Another example presented earlier was a regex for finding the filenames in HTML image tags. Here's how we would compile the regex, adding flags so that it's not case-sensitive, and allowing us to include comments:

image_re = re.compile(r"""
                <img\s+                  # start of tag
                [^>]*?                   # non-src attributes
                src=                     # start of src attribute
                (?P<quote>["'])?         # optional opening quote
                (?P<image>[^"'>]+)       # image filename
                (?(quote)(?P=quote))     # closing quote
                [^>]*?                   # non-src attributes
                >                        # end of the tag
                 """, re.IGNORECASE|re.VERBOSE)
image_files = []
for match in image_re.finditer(text):
    image_files.append(match.group("image"))

Again, we use the finditer() method to retrieve each match, and the match object's group() function to retrieve the captured texts. Since the case-insensitivity applies only to img and src, we could drop the re.IGNORECASE flag and use [Ii][Mm][Gg] and [Ss][Rr][Cc] instead. Although this design would make the regex less clear, it might be faster, because it wouldn't require the text being matched to be set to uppercase or lowercase. But it's likely to make a difference only if the regex is being used on a very large amount of text.

One common task is to take an HTML text and output just the plain text that it contains. Naturally, we could do this using one of Python's parsers, but a simple tool can be created using regexes. Three tasks need to be done: delete any tags, replace entities with the characters they represent, and insert blank lines to separate paragraphs. Here's a function (taken from the html2text.py program) that does the job:

def html2text(html_text):
    def char_from_entity(match):
        code = html.entities.name2codepoint.get(match.group(1), 0xFFFD)
        return chr(code)
    text = re.sub(r"<!--(?:.|\n)*?-->", "", html_text)          #1
    text = re.sub(r"<[Pp][^>]*?(?!</)>", "\n\n", text)          #2
    text = re.sub(r"<[^>]*?>", "", text)                        #3
    text = re.sub(r"&#(\d+);", lambda m: chr(int(m.group(1))), text)
    text = re.sub(r"&([A-Za-z]+);", char_from_entity, text)     #5
    text = re.sub(r"\n(?:[ \xA0\t]+\n)+", "\n", text)           #6
    return re.sub(r"\n\n+", "\n\n", text.strip())               #7

The first regex matches HTML comments, including those with other HTML tags nested inside them:

<!--(?:.|\n)*?-->

The re.sub() function replaces as many matches as it finds with the replacement—deleting the matches if the replacement is an empty string, as it is here. (We can specify a maximum number of matches by giving an additional integer argument at the end.)

We're careful to use nongreedy (minimal) matching to ensure that we delete one comment for each match; without this, we'd delete from the start of the first comment to the end of the last comment.

The re.sub() function doesn't accept any flags as arguments; . means "any character except newline," so we must look for . or \n. And we must look for these using alternation rather than a character class, since inside a character class . has its literal meaning (period). An alternative would be to begin the regex with the flag embedded:

(?s)<!--.*?-->

Or we could compile a regex object with the re.DOTALL flag, in which case the regex would simply be as follows:

<!--.*?-->

The second regex in the example matches opening paragraph tags (such as <P> or <p align=center>):

<[Pp][^>]*?(?!</)>

It matches the opening <p (or <P), any attributes (using nongreedy matching), and finally the closing >, providing that it isn't preceded by / (using a negative lookbehind assertion), since that would indicate a closing paragraph tag. The second call to the re.sub() function uses this regex to replace opening paragraph tags with two newline characters (the standard way to delimit a paragraph in a plaintext file).

The third regex matches any tag and is used in the third re.sub() call to delete all the remaining tags:

<[^>]*?>

HTML entities are a way of specifying non-ASCII characters by using ASCII characters. They come in two forms:

• &name; where name is the name of the character. For example, &copy; for the copyright symbol (©).
• &#digits; where digits are decimal digits identifying the Unicode code point. For example, &#165; for the yen symbol (¥).

The fourth call to re.sub() uses the following regex, which matches the digits form and captures the digits into capture group 1:

&#(\d+);

Instead of a literal replacement text, we've passed a lambda function. When a function is passed to re.sub(), it calls the function once for each time it matches, passing the match object as the function's sole argument. Inside the lambda function, we retrieve the digits (as a string), convert to an integer by using the built-in int() function, and then use the built-in chr() function to obtain the Unicode character for the given code point. The function's return value (or, in the case of a lambda expression, the result of the expression) is used as the replacement text.

The fifth re.sub() call uses the following regex to capture named entities:

&([A-Za-z]+);

The standard library's html.entities module contains dictionaries of entities, including name2codepoint, whose keys are entity names and whose values are integer code points. The re.sub() function calls the local char_from_entity() function every time it has a match. The char_from_entity() function uses dict.get() with a default argument of 0xFFFD (the code point of the standard Unicode replacement character—often depicted as ?). This ensures that a code point is always retrieved, and it's used with the chr() function to return a suitable character to replace the named entity—using the Unicode replacement character if the entity name is invalid.

The sixth re.sub() call's regex is used to delete lines that contain only whitespace:

\n(?:[ \xA0\t]+\n)+

The character class we've used contains a space, a nonbreaking space (with which &nbsp; entities are replaced in the preceding regex), and a tab. The regex matches a newline (the one at the end of a line that precedes one or more whitespace-only lines), at least one line (and as many lines as possible) that contain(s) only whitespace. Since the match includes the newline, from the line preceding the whitespace-only lines we must replace the match with a single newline; otherwise, we would delete not just the whitespace-only lines but also the newline of the line that preceded them.

The result of the seventh and last re.sub() call is returned to the caller:

\n\n+

This regex is used to replace sequences of two or more newlines with exactly two newlines, to ensure that paragraphs are separated by just one blank line.

In the HTML example, none of the replacements were taken directly from the match (although HTML entity names and numbers were used), but in some situations the replacement might need to include all or some of the matching text. For example, if we have a list of names, each of the form Forename Middlename1...MiddlenameN Surname, where there may be any number of middle names (including none), and we want to produce a new version of the list with each item of the form Surname,ForenameMiddlename1...MiddlenameN, we can easily do so using a regex:

new_names = []
for name in names:
    name = re.sub(r"(\w+(?:\s+\w+)*)\s+(\w+)", r"\2, \1", name)
    new_names.append(name)

The first part of the regex matches the forename with the first \w+ expression and zero or more middle names with the (?:\s+\w+)* expression:

(\w+(?:\s+\w+)*)

The middle name expression matches zero or more occurrences of whitespace followed by a word.

The second part of the regex matches the whitespace that follows the forename (and middle names) and the surname:

\s+(\w+)

If this regex looks a bit too much like line noise, we can use named capture groups to improve legibility and make it more maintainable:

name = re.sub(r"(?P<forenames>\w+(?:\s+\w+)*)"
              r"\s+(?P<surname>\w+)",
              r"\g<surname>, \g<forenames>", name)

Captured text can be referred to in a sub() or subn() function or method by using the syntax \i or \g<id>, where i is the number of the capture group and id is the name or number of the capture group—so \1 is the same as \g<1>; in this example, the same as \g<forenames>. This syntax can also be used in the string passed to a match object's expand() method.

Why doesn't the first part of the regex grab the entire name? After all, it's using greedy matching. In fact, it will, but then the match will fail because although the middle names part can match zero or more times, the surname part must match exactly once, but the greedy middle names part has grabbed everything. Having failed, the regular expression engine will then backtrack, giving up the last "middle name" and thus allowing the surname to match.

Although greedy matches match as much as possible, they stop if matching more would make the match fail. For example, if the name is James W. Loewen, the regex will first match the entire name; that is, James W. Loewen. This satisfies the first part of the regex but leaves nothing for the surname part to match, and since the surname is mandatory (it has an implicit quantifier of 1), the regex has failed. Since the middle names part is quantified by *, it can match zero or more times (currently it's matching twice, " W." and " Loewen"), so the regular expression engine can make it give up some of its match without causing it to fail. Therefore, the regex backtracks, giving up the last \s+\w+ (" Loewen"), so the match becomes James W. Loewen, with the match satisfying the whole regex and with the two match groups containing the correct texts.

When we use alternation (|) with two or more alternatives capturing, we don't know which alternative matched, so we don't know from which capture group to retrieve the captured text. Of course, we can iterate over all the groups to find the nonempty one, but quite often in this situation the match object's lastindex attribute can give us the number of the group we want. We'll look at one last example to illustrate this principle and to give us a little more regex practice.

Suppose we want to find out what encoding an HTML, XML, or Python file is using. We could open the file in binary mode and read, say, the first 1000 bytes into a bytes object. Then we could close the file, look for an encoding in the bytes object, and reopen the file in text mode using the encoding we found or using a fallback encoding (such as UTF-8). The regex engine expects regexes to be supplied as strings, but the text to which the regex is applied can be a str, bytes, or bytearray object. When bytes or bytearray objects are used, all the functions and methods return bytes instead of strings, and the re.ASCII flag is implicitly switched on.

For HTML files, the encoding is normally specified in a <meta> tag (if specified at all):

<meta http-equiv='Content-Type' content='text/html; charset=ISO-8859-1'/>

XML files are UTF-8 by default, but this can be overridden:

<?xml version="1.0" encoding="Shift_JIS"?>

Python 3 files are also UTF-8 by default, but again this can be overridden by including a line such as the following:

# encoding: latin1

or this one, immediately after the shebang line:

# -*- coding: latin1 -*-

Here's how we would find the encoding, assuming that the variable binary is a bytes object containing the first 1000 bytes of an HTML, XML, or Python file:

match = re.search(r"""(?<![-\w])                    #1
                      (?:(?:en)?coding|charset)     #2
                      (?:=(["'])?([-\w]+)(?(1)\1)   #3
                      |:\s*([-\w]+))""".encode("utf8"),
                  binary, re.IGNORECASE|re.VERBOSE)
encoding = match.group(match.lastindex) if match else b"utf8"

To search a bytes object, we must specify a pattern that is also a bytes object. In this case, we want the convenience of using a raw string, so we use one and convert it to a bytes object as the re.search() function's first argument.

The first part of the regex itself is a lookbehind assertion that says that the match cannot be preceded by a hyphen or a word character. The second part matches encoding, coding, or charset and could have been written as follows:

(?:encoding|coding|charset)

We've made the third part span two lines to emphasize the fact that it has two alternating parts, only one of which can match:

=(["'])?([-\w]+)(?(1)\1)

:\s*([-\w]+)

The first of these matches an equal sign followed by one or more word or hyphen characters (optionally enclosed in matching quotes using a conditional match). The second matches a colon and then optional whitespace followed by one or more word or hyphen characters. (Remember that a hyphen inside a character class is taken to be a literal hyphen if it's the first character. Otherwise, it means a range of characters; for example, [0-9].)

We've used the re.IGNORECASE flag to avoid having to write the following:

(?:(?:[Ee][Nn])?[Cc][Oo][Dd][Ii][Nn][Gg]|[Cc][Hh][Aa][Rr][Ss][Ee][Tt])

We've used the re.VERBOSE flag so that we can lay out the regex neatly and include comments (in this case, just numbers to make the parts easy to refer to in this text).

There are three capturing match groups, all in the third part:

• (["'])? captures the optional opening quote.
• ([-\w]+) captures an encoding that follows an equal sign.
• The second ([-\w]+) on the following line captures an encoding that follows a colon.

We're only interested in the encoding, so we want to retrieve either the second or third capture group, only one of which can match, since they're alternatives. The lastindex attribute holds the index of the last matching capture group (either 2 or 3 when a match occurs in this example), so we retrieve whichever matched, or use a default encoding if no match was made.

We've seen all of the most frequently used re module functionality in action, so we'll conclude this section by mentioning one last function. The re.split() function, or the regex object's split() method, can split strings based on a regex. One common requirement is to split text on whitespace to get a list of words. This can be done using re.split(r"\s+", text), which returns a list of words (or, more precisely, a list of strings, each of which matches \S+).

Regular expressions are very powerful and useful; once you've learned how to use regexes, it's easy to see all text problems as requiring a regex solution. But sometimes using string methods is both sufficient and more appropriate. For example, we can just as easily split on whitespace by using text.split(), since the str.split() method's default behavior (or with a first argument of None) is to split on \s+.