Text and String Methods (str accessor and regex)

Why not just use Python string methods?

Python’s built-in string methods (.lower(), .split(), .replace()) work great on a single string. But a DataFrame column can hold thousands of strings — and, almost always, a few missing values mixed in. Calling .lower() directly on every value with .map() works fine until it hits a NaN, and then it crashes.

Pandas solves this with the str accessor: series.str.lower() runs a vectorized, element-wise version of the string method across the whole column, and automatically skips (propagates) missing values instead of erroring out.

Example dataset

Messy text column

import pandas as pd
 
emails = pd.Series(
    ["Dave@Google.com", "steve@gmail.com ", None, "  rob@gmail.com"],
    index=["Dave", "Steve", "Wes", "Rob"],
)
 
print(emails)
print(emails.isna())

Everyday cleanup: lower, strip, contains, replace

lower / strip / title

print(emails.str.lower())
print(emails.str.strip().str.lower())

str.contains checks each value for a substring (or pattern) and returns a boolean Series — pass na=False so missing values count as “no match” instead of showing up as NaN in a mask:

str.contains with na=False

has_gmail = emails.str.contains("gmail", case=False, na=False)
print(has_gmail)
print(emails[has_gmail])

str.replace swaps out a substring everywhere it appears — pass regex=False when your search text isn’t a pattern, so characters like . aren’t treated as “any character”:

str.replace (literal, not regex)

print(emails.str.replace("gmail", "GMAIL", regex=False))

Splitting and pulling pieces out

str.split breaks each string into a list at a delimiter; indexing into the result with .str[i] or .str.get(i) pulls out one piece of each list:

str.split and str.get

parts = emails.str.strip().str.split("@")
print(parts)
print(parts.str.get(0))   # username, before the @
print(parts.str.get(1))   # domain, after the @

Slicing works too, the same way you’d slice a plain Python string:

Slicing with the str accessor

print(emails.str.strip().str[:4])

Regex with `.str.extract` and `.str.findall`

For real pattern matching — not just a fixed substring — pass a regular expression. str.extract pulls out groups (the parts in parentheses) into their own columns; str.findall returns every match as a list:

str.extract splits an email into groups

import re
 
pattern = r"([A-Za-z0-9._%+-]+)@([A-Za-z0-9.-]+)\.([A-Za-z]{2,4})"
 
clean_emails = emails.str.strip()
print(clean_emails.str.extract(pattern))

The result is a small DataFrame with one column per capture group — column 0 is the username, 1 is the domain, 2 is the suffix. Rows that don’t match (or were NaN to begin with) come back as NaN in every column, not an error.

str.findall returns every match

print(clean_emails.str.findall(pattern, flags=re.IGNORECASE))

Common pitfalls

Forgetting regex=False on str.replace when your replacement text contains regex special characters like . or ( — it will silently match more than you intended.
Boolean masks from str.contains can contain NaN (for missing input) unless you pass na=False — filtering with a mask that still has NaN in it raises an error.
str.extract needs parentheses (...) around the parts you want back; a pattern with no groups extracts nothing useful.

Visualize it

diagram The str accessor skips missing values mermaid

  flowchart LR
  A["Series with NaN mixed in"] --> B[".str accessor"]
  B --> C["Real values -> string/regex logic"]
  B --> D["NaN -> stays NaN"]
  C --> E["Cleaned Series"]
  D --> E

Every str method runs element-wise down a column, passing real values through the string/regex logic and letting NaN pass straight through untouched.

🧪 Try It Yourself

Exercise 1 – Lowercase an Entire Column

Exercise 2 – Check Membership While Ignoring NaN

Exercise 3 – Extract a Regex Group

With text under control, move on to Reading and Writing Data (CSV, Excel, JSON) to load real datasets that will need exactly this kind of cleanup.

pch.coffeeTagline

pch.coffeeCta

pch.feedbackHeading

pch.feedbackSubheading