Regular Expressions

Why Work with Text?¶

In this note, we’ll discuss the necessary tools to manipulate text: Python string manipulation and regular expressions.

There are two main reasons for working with text.

1. Canonicalization: Convert data that has multiple formats into a standard form.

   • By manipulating text, we can join tables with mismatched string labels.

2. Extract information into a new feature.

   • For example, we can extract date and time features from text.

Python String Methods¶

First, we’ll introduce a few methods useful for string manipulation. The following table includes a number of string operations supported by Python and polars. The Python functions operate on a single string, while their equivalent in polars are vectorized — they operate on a Series of string data.

We’ll discuss the differences between Python string functions and Polars .str methods in the following section on canonicalization.

Canonicalization¶

Assume we want to merge the given tables.

import polars as pl

with open('data/county_and_state.csv') as f:
    county_and_state = pl.read_csv(f)
    
with open('data/county_and_population.csv') as f:
    county_and_pop = pl.read_csv(f)

Can we convert these columns into one standard, canonical form to merge the two tables?

Canonicalization with Python String Manipulation¶

The following function uses Python string manipulation to convert a single county name into canonical form. It does so by eliminating whitespace, punctuation, and unnecessary text.

Canonicalization with Polars Series Methods¶

Alternatively, we can use Polars .str methods to create this standardized column. These operations are vectorized, so they can be applied to an entire Series without manually looping over individual strings. To do so, we access the .str namespace of the Series before calling string methods such as .to_lowercase() and .replace(). Polars string method names are similar to, but not always identical to, Python’s built-in string methods.

Chaining multiple Series methods in this manner eliminates the need to use the map function (as this code is vectorized).

Extraction¶

Extraction explores the idea of obtaining useful information from text data. This will be particularly important in model building, which we’ll study in a few weeks.

Say we want to read some data from a .txt file.

Suppose we want to extract the day, month, year, hour, minutes, seconds, and time zone. Unfortunately, these items are not in a fixed position from the beginning of the string, so slicing by some fixed offset won’t work.

Instead, we can use some clever thinking. Notice how the relevant information is contained within a set of brackets, further separated by / and :. We can hone in on this region of text, and split the data on these characters. Python’s built-in .split function makes this easy.

There are two problems with this code:

1. Python’s built-in functions limit us to extract data one record at a time,

2. The code is quite verbose.

   • This is a larger issue that is trickier to solve

In the next section, we’ll introduce regular expressions - a tool that solves problem 2.

RegEx Basics¶

A regular expression (“RegEx”) is a sequence of characters that specifies a search pattern. They are written to extract specific information from text. Regular expressions are essentially part of a smaller programming language embedded in Python, made available through the re module. As such, they have a stand-alone syntax and methods for various capabilities.

Regular expressions are useful in many applications beyond data science. For example, Social Security Numbers (SSNs) are often validated with regular expressions.

There are a ton of resources to learn and experiment with regular expressions. A few are provided below:

• Official Regex Guide

• Regex101.com

  • Be sure to check Python under the category on the left.

Basic RegEx Syntax¶

There are four basic operations with regular expressions.

Notice how these metacharacter operations are ordered. Rather than being literal characters, these metacharacters manipulate adjacent characters. () takes precedence, followed by *, and finally |. This allows us to differentiate between very different regex commands like AB* and (AB)*. The former reads “A then zero or more copies of B”, while the latter specifies “zero or more copies of AB”.

Examples¶

Question 1: Give a regular expression that matches moon, moooon, etc. Your expression should match any even number of os except zero (i.e. don’t match mn).

Question 2: Using only basic operations, formulate a regex that matches muun, muuuun, moon, moooon, etc. Your expression should match any even number of us or os except zero (i.e. don’t match mn).

RegEx Expanded¶

Provided below are more complex regular expression functions.

A character class matches a single character in its class. These characters can be hardcoded —— in the case of [aeiou] —— or shorthand can be specified to mean a range of characters. Examples include:

1. [A-Z]: Any capitalized letter

2. [a-z]: Any lowercase letter

3. [0-9]: Any single digit

4. [A-Za-z]: Any capitalized or lowercase letter

5. [A-Za-z0-9]: Any capitalized or lowercase letter or single digit

Examples¶

Let’s analyze a few examples of complex regular expressions.

Explanations

1. .*SPB.* only matches strings that contain the substring SPB.

   • The .* metacharacter matches any amount of non-negative characters. Newlines do not count.

2. This regular expression matches 3 of any digit, then a dash, then 2 of any digit, then a dash, then 4 of any digit.

   • You’ll recognize this as the familiar Social Security Number regular expression.

3. Matches any email with a com or edu domain, where all characters of the email are letters.

   • At least one . must precede the domain name. Including a backslash \ before any metacharacter (in this case, the .) tells RegEx to match that character exactly.

Convenient RegEx¶

Here are a few more convenient regular expressions.

Greediness¶

In order to fully understand the last operation in the table, we have to discuss greediness. RegEx is greedy – it will look for the longest possible match in a string. By default, RegEx quantifiers such as *, +, and {} are greedy. That means they will look for the longest possible match in a string such that the rest of the pattern can still match. Anywhere a quantifier can “cut repetition short,” greediness becomes relevant.

To motivate this with an example, consider the pattern <div>.*</div>. In the sentence below, we would hope that the two bolded portions would be matched separately:

“This is a <div>example</div> of greediness <div>in</div> regular expressions.”

However, in reality, RegEx captures far more of the sentence. The way RegEx processes the text given that pattern is as follows:

1. “Look for the exact string <div>”

2. Then, “look for any character 0 or more times"

3. Then, “look for the exact string </div>"

The result would be all the characters starting from the leftmost <div> and the rightmost </div> (inclusive):

“This is a <div>example</div> of greediness <div>in</div> regular expressions.”

We can fix this by making our pattern non-greedy by adding a ? after the quantifier, <div>.*?</div>. Similarly, +? is the non-greedy version of +. Here, ? modifies the quantifier, which is a different use of ? from the “zero or one” quantifier. You can read up more in the documentation here. If you are still confused, Regex 101 can be a very effective tool for trying out different strings and testing how patterns behave.

Examples¶

Let’s revisit our earlier problem of extracting date/time data from the given .txt files. Here is how the data looked.

Question: Give a regular expression that matches everything contained within and including the brackets - the day, month, year, hour, minutes, seconds, and time zone.

Regex in Python and Polars (RegEx Groups)¶

Canonicalization¶

Canonicalization with RegEx¶

Earlier in this note, we examined the process of canonicalization using python string manipulation and polars Series methods. However, we mentioned this approach had a major flaw: our code was unnecessarily verbose. Equipped with our knowledge of regular expressions, let’s fix this.

To do so, we need to understand a few functions in the re module. The first of these is the substitute function: re.sub(pattern, repl, text). It behaves similarly to python’s built-in .replace function, and returns text with all instances of pattern replaced by repl.

The regular expression here removes text surrounded by <> (also known as HTML tags).

In order, the pattern matches ...

1. a single <

2. any character that is not a > : div, td valign..., /td, /div

3. a single >

Any substring in text that fulfills all three conditions will be replaced by ''.

Notice the r preceding the regular expression pattern; this specifies the regular expression is a raw string. Raw strings do not recognize escape sequences (i.e., the Python newline metacharacter \n). This makes them useful for regular expressions, which often contain literal \ characters.

In other words, don’t forget to tag your RegEx with an r.

Canonicalization with polars¶

We can also use regular expressions with polars Series methods. This gives us the benefit of operating on an entire column of data as opposed to a single value. The code is simple:
ser.str.replace_all(pattern, repl).

Consider the following DataFrame html_data with a single column.

Extraction¶

Extraction with RegEx¶

Just like with canonicalization, the re module provides capability to extract relevant text from a string:
re.findall(pattern, text). This function returns a list of all matches to pattern.

Using the familiar regular expression for Social Security Numbers:

Extraction with polars¶

polars provides extraction functionality on a Series of data using ser.str.extract_all(pattern).

Consider the following DataFrame ssn_data.

This function returns a list for every row containing the pattern matches in a given string.

For capture groups, polars provides ser.str.extract_groups(pattern). To expand the captured groups into separate columns, follow it with .struct.unnest().

Regular Expression Capture Groups¶

Earlier we used parentheses ( ) to specify the highest order of operation in regular expressions. However, they have another meaning; parentheses are often used to represent capture groups. Capture groups are essentially, a set of smaller regular expressions that match multiple substrings in text data.

Let’s take a look at an example.

Example 1¶

Say we want to capture all occurrences of time data (hour, minute, and second) as separate entities.

Notice how the given pattern has 3 capture groups, each specified by the regular expression (\d\d). We then use re.findall to return these capture groups, each as tuples containing 3 matches.

These regular expression capture groups can be different. We can use the (\d{2}) shorthand to extract the same data.

Example 2¶

With the notion of capture groups, convince yourself how the following regular expression works.

String Function Summary¶

Limitations of Regular Expressions¶

Today, we explored the capabilities of regular expressions in data wrangling with text data. However, there are a few things to be wary of.

Writing regular expressions is like writing a program.

• Need to know the syntax well.

• Can be easier to write than to read.

• Can be difficult to debug.

Regular expressions are terrible at certain types of problems:

• For parsing a hierarchical structure, such as JSON, use the json.load() parser, not regex.

• For parsing real-world HTML/XML, use an appropriate parser such as html.parser rather than regex.

• Regex cannot handle tasks such as counting the same number of instances of two different patterns.

LLMs can sometimes solve similar text-parsing tasks, but they can be less reliable and more computationally expensive. For clear, well-defined patterns, regex is often the better tool; more complex text tasks may be better suited to an LLM.

Ultimately, the goal is not to memorize all regular expressions. Rather, the aim is to:

• Understand what RegEx is capable of.

• Parse and create RegEx, with a reference table

• Use vocabulary (metacharacter, escape character, groups, etc.) to describe regex metacharacters.

• Differentiate between (), [], {}

• Design your own character classes with \d, \w, \s, […-…], ^, etc.

• Use Python re functions and Polars .str methods.