What "dirty data" looks like
Common problems:
- Numbers stored as strings (
"1,200"," 42 ") - Inconsistent categories (
"Delhi","delhi"," DELHI ") - Duplicate rows
- Weird whitespace and punctuation
Example dataset
import pandas as pd
df = pd.DataFrame({
"name": [" Asha ", "Ravi", "ravi", "Meera"],
"city": ["delhi", "Delhi ", "DELHI", " Pune"],
"amount": ["1,200", "500", "500", " 700 "],
})
print(df)1) Fix text: str.strip, str.lower, str.replace
clean = df.copy()
clean["name"] = clean["name"].str.strip()
clean["city"] = clean["city"].str.strip().str.lower()
print(clean)2) Convert string numbers to numeric: to_numeric
Remove commas and spaces first
clean = df.copy()
clean["amount"] = clean["amount"].astype(str).str.replace(",", "", regex=False).str.strip()
clean["amount"] = pd.to_numeric(clean["amount"], errors="coerce")
print(clean)
print(clean.dtypes)errors="coerce" turns invalid values into NaN (so you can handle them with dropna/fillna).
3) Change column types safely: astype
clean = clean.astype({"amount": "int64"})
print(clean.dtypes)If conversion fails, Pandas throws an error. Prefer to_numeric(..., errors="coerce") for messy data.
4) Remove duplicates: drop_duplicates
Exact duplicates
dedup = clean.drop_duplicates()
print(dedup)Duplicates based on a subset of columns
dedup = clean.drop_duplicates(subset=["name", "city"], keep="first")
print(dedup)Tip: validate your cleaning
After cleaning, quickly check:
df.dtypesdf.isna().sum()df["col"].value_counts()for categories
Cleaning is not a single step. It's a loop: inspect → clean → validate.
Visualize it
flowchart LR A["Inspect (dtypes, head)"] --> B["Clean (strip, replace, astype)"] B --> C["Validate (isna, value_counts)"] C -->|"still messy"| A C -->|"looks good"| D["Ready for analysis"]
🧪 Try It Yourself
Exercise 1 – Strip Whitespace From Text
Tab indents. Press Escape, then Tab, to leave the editor. Control plus Enter runs the code; Control plus Shift plus Enter submits it.
Press Run to see what your code prints.Exercise 2 – Convert Text Numbers Safely
Tab indents. Press Escape, then Tab, to leave the editor. Control plus Enter runs the code; Control plus Shift plus Enter submits it.
Press Run to see what your code prints.Exercise 3 – Drop Duplicate Rows
Tab indents. Press Escape, then Tab, to leave the editor. Control plus Enter runs the code; Control plus Shift plus Enter submits it.
Press Run to see what your code prints.Check your understanding
What does pd.to_numeric(col, errors="coerce") do with a value that cannot be parsed as a number?
errors=coerce converts invalid values to NaN so you can handle them with dropna or fillna. astype, by contrast, raises an error.
Show answer
B — Turns it into NaN — errors=coerce converts invalid values to NaN so you can handle them with dropna or fillna. astype, by contrast, raises an error.
To convert the amount strings "1,200" and " 700 " to numbers, what must you do before to_numeric?
The lesson chains str.replace(",", "", regex=False) and str.strip() first, then calls to_numeric.
Show answer
A — Remove the commas and strip the spaces — The lesson chains str.replace(",", "", regex=False) and str.strip() first, then calls to_numeric.
Which call removes rows that repeat the same name and city, keeping the first one?
drop_duplicates with subset compares only those columns. duplicated().sum() just counts exact duplicate rows.
Show answer
A — df.drop_duplicates(subset=["name", "city"], keep="first") — drop_duplicates with subset compares only those columns. duplicated().sum() just counts exact duplicate rows.
Why does the lesson normalize city values with str.strip().str.lower() before comparing?
Inconsistent case and stray whitespace create fake categories and hide duplicates.
Show answer
A — Delhi, delhi and DELHI would otherwise count as different categories — Inconsistent case and stray whitespace create fake categories and hide duplicates.
Next
Text cleanup is worth its own deep dive — continue to Text and String Methods (str
accessor and regex) to handle messier text patterns with the .str accessor and regex.