Regular Expressions

Why Work with Text?¶

Last lecture, we learned of the difference between quantitative and qualitative variable types. The latter includes string data — the primary focus of lecture 6. In this note, we’ll discuss the necessary tools to manipulate text: Python string manipulation and regular expressions.

There are two main reasons for working with text.

1. Canonicalization: Convert data that has multiple formats into a standard form.

   • By manipulating text, we can join tables with mismatched string labels.

2. Extract information into a new feature.

   • For example, we can extract date and time features from text.

Python String Methods¶

First, we’ll introduce a few methods useful for string manipulation. The following table includes a number of string operations supported by Python and pandas. The Python functions operate on a single string, while their equivalent in pandas are vectorized — they operate on a Series of string data.

We’ll discuss the differences between Python string functions and pandas Series methods in the following section on canonicalization.

Canonicalization¶

Assume we want to merge the given tables.

import pandas as pd

with open('data/county_and_state.csv') as f:
    county_and_state = pd.read_csv(f)
    
with open('data/county_and_population.csv') as f:
    county_and_pop = pd.read_csv(f)

Can we convert these columns into one standard, canonical form to merge the two tables?

Canonicalization with Python String Manipulation¶

The following function uses Python string manipulation to convert a single county name into canonical form. It does so by eliminating whitespace, punctuation, and unnecessary text.

Canonicalization with Pandas Series Methods¶

Alternatively, we can use pandas Series methods to create this standardized column. To do so, we must call the .str attribute of our Series object prior to calling any methods, like .lower and .replace. Notice how these method names match their equivalent built-in Python string functions.

Chaining multiple Series methods in this manner eliminates the need to use the map function (as this code is vectorized).

Extraction¶

Extraction explores the idea of obtaining useful information from text data. This will be particularily important in model building, which we’ll study in a few weeks.

Say we want to read some data from a .txt file.

Suppose we want to extract the day, month, year, hour, minutes, seconds, and time zone. Unfortunately, these items are not in a fixed position from the beginning of the string, so slicing by some fixed offset won’t work.

Instead, we can use some clever thinking. Notice how the relevant information is contained within a set of brackets, further separated by / and :. We can hone in on this region of text, and split the data on these characters. Python’s built-in .split function makes this easy.

There are two problems with this code:

1. Python’s built-in functions limit us to extract data one record at a time,

2. The code is quite verbose.

   • This is a larger issue that is trickier to solve

In the next section, we’ll introduce regular expressions - a tool that solves problem 2.

RegEx Basics¶

A regular expression (“RegEx”) is a sequence of characters that specifies a search pattern. They are written to extract specific information from text. Regular expressions are essentially part of a smaller programming language embedded in Python, made available through the re module. As such, they have a stand-alone syntax and methods for various capabilities.

Regular expressions are useful in many applications beyond data science. For example, Social Security Numbers (SSNs) are often validated with regular expressions.

There are a ton of resources to learn and experiment with regular expressions. A few are provided below:

• Official Regex Guide

• Data 100 Reference Sheet

• Regex101.com

  • Be sure to check Python under the category on the left.

Basics RegEx Syntax¶

There are four basic operations with regular expressions.

Notice how these metacharacter operations are ordered. Rather than being literal characters, these metacharacters manipulate adjacent characters. () takes precedence, followed by *, and finally |. This allows us to differentiate between very different regex commands like AB* and (AB)*. The former reads “A then zero or more copies of B”, while the latter specifies “zero or more copies of AB”.

Examples¶

Question 1: Give a regular expression that matches moon, moooon, etc. Your expression should match any even number of os except zero (i.e. don’t match mn).

Question 2: Using only basic operations, formulate a regex that matches muun, muuuun, moon, moooon, etc. Your expression should match any even number of us or os except zero (i.e. don’t match mn).

RegEx Expanded¶

Provided below are more complex regular expression functions.

A character class matches a single character in its class. These characters can be hardcoded —— in the case of [aeiou] —— or shorthand can be specified to mean a range of characters. Examples include:

1. [A-Z]: Any capitalized letter

2. [a-z]: Any lowercase letter

3. [0-9]: Any single digit

4. [A-Za-z]: Any capitalized or lowercase letter

5. [A-Za-z0-9]: Any capitalized or lowercase letter or single digit

Examples¶

Let’s analyze a few examples of complex regular expressions.

1. .*SPB.* only matches strings that contain the substring SPB.

   • The .* metacharacter matches any amount of non-negative characters. Newlines do not count.

2. This regular expression matches 3 of any digit, then a dash, then 2 of any digit, then a dash, then 4 of any digit.

   • You’ll recognize this as the familiar Social Security Number regular expression.

3. Matches any email with a com or edu domain, where all characters of the email are letters.

   • At least one . must precede the domain name. Including a backslash \ before any metacharacter (in this case, the .) tells RegEx to match that character exactly.

Convenient RegEx¶

Here are a few more convenient regular expressions.

Greediness¶

In order to fully understand the last operation in the table, we have to discuss greediness. RegEx is greedy – it will look for the longest possible match in a string. To motivate this with an example, consider the pattern <div>.*</div>. In the sentence below, we would hope that the bolded portions would be matched:

“This is a <div>example</div> of greediness <div>in</div> regular expressions.”

However, in reality, RegEx captures far more of the sentence. The way RegEx processes the text given that pattern is as follows:

1. “Look for the exact string <\div>”

2. Then, “look for any character 0 or more times"

3. Then, “look for the exact string </div>"

The result would be all the characters starting from the leftmost <div> and the rightmost </div> (inclusive):

“This is a <div>example</div> of greediness <div>in</div> regular expressions.”

We can fix this by making our pattern non-greedy, <div>.*?</div>. You can read up more in the documentation here.

Examples¶

Let’s revisit our earlier problem of extracting date/time data from the given .txt files. Here is how the data looked.

Question: Give a regular expression that matches everything contained within and including the brackets - the day, month, year, hour, minutes, seconds, and time zone.

Regex in Python and Pandas (RegEx Groups)¶

Canonicalization¶

Canonicalization with RegEx¶

Earlier in this note, we examined the process of canonicalization using python string manipulation and pandas Series methods. However, we mentioned this approach had a major flaw: our code was unnecessarily verbose. Equipped with our knowledge of regular expressions, let’s fix this.

To do so, we need to understand a few functions in the re module. The first of these is the substitute function: re.sub(pattern, rep1, text). It behaves similarly to python’s built-in .replace function, and returns text with all instances of pattern replaced by rep1.

The regular expression here removes text surrounded by <> (also known as HTML tags).

In order, the pattern matches ...

1. a single <

2. any character that is not a > : div, td valign..., /td, /div

3. a single >

Any substring in text that fulfills all three conditions will be replaced by ''.

Notice the r preceding the regular expression pattern; this specifies the regular expression is a raw string. Raw strings do not recognize escape sequences (i.e., the Python newline metacharacter \n). This makes them useful for regular expressions, which often contain literal \ characters.

In other words, don’t forget to tag your RegEx with an r.

Canonicalization with pandas¶

We can also use regular expressions with pandas Series methods. This gives us the benefit of operating on an entire column of data as opposed to a single value. The code is simple:
ser.str.replace(pattern, repl, regex=True).

Consider the following DataFrame html_data with a single column.

Extraction¶

Extraction with RegEx¶

Just like with canonicalization, the re module provides capability to extract relevant text from a string:
re.findall(pattern, text). This function returns a list of all matches to pattern.

Using the familiar regular expression for Social Security Numbers:

Extraction with pandas¶

pandas similarily provides extraction functionality on a Series of data: ser.str.findall(pattern)

Consider the following DataFrame ssn_data.

This function returns a list for every row containing the pattern matches in a given string.

As you may expect, there are similar pandas equivalents for other re functions as well. Series.str.extract takes in a pattern and returns a DataFrame of each capture group’s first match in the string. In contrast, Series.str.extractall returns a multi-indexed DataFrame of all matches for each capture group. You can see the difference in the outputs below:

Regular Expression Capture Groups¶

Earlier we used parentheses ( ) to specify the highest order of operation in regular expressions. However, they have another meaning; parentheses are often used to represent capture groups. Capture groups are essentially, a set of smaller regular expressions that match multiple substrings in text data.

Let’s take a look at an example.

Example 1¶

Say we want to capture all occurences of time data (hour, minute, and second) as separate entities.

Notice how the given pattern has 3 capture groups, each specified by the regular expression (\d\d). We then use re.findall to return these capture groups, each as tuples containing 3 matches.

These regular expression capture groups can be different. We can use the (\d{2}) shorthand to extract the same data.

Example 2¶

With the notion of capture groups, convince yourself how the following regular expression works.

Limitations of Regular Expressions¶

Today, we explored the capabilities of regular expressions in data wrangling with text data. However, there are a few things to be wary of.

Writing regular expressions is like writing a program.

• Need to know the syntax well.

• Can be easier to write than to read.

• Can be difficult to debug.

Regular expressions are terrible at certain types of problems:

• For parsing a hierarchical structure, such as JSON, use the json.load() parser, not RegEx!

• Complex features (e.g. valid email address).

• Counting (same number of instances of a and b). (impossible)

• Complex properties (palindromes, balanced parentheses). (impossible)

Ultimately, the goal is not to memorize all regular expressions. Rather, the aim is to:

• Understand what RegEx is capable of.

• Parse and create RegEx, with a reference table

• Use vocabulary (metacharacter, escape character, groups, etc.) to describe regex metacharacters.

• Differentiate between (), [], {}

• Design your own character classes with \d, \w, \s, […-…], ^, etc.

• Use python and pandas RegEx methods.