Cleaning Data (astype, duplicates, string cleaning)

What “dirty data” looks like

Common problems:

Numbers stored as strings ("1,200", " 42 ")
Inconsistent categories ("Delhi", "delhi", " DELHI ")
Duplicate rows
Weird whitespace and punctuation

Example dataset

Dirty data

import pandas as pd
 
df = pd.DataFrame({
    "name": [" Asha ", "Ravi", "ravi", "Meera"],
    "city": ["delhi", "Delhi ", "DELHI", " Pune"],
    "amount": ["1,200", "500", "500", " 700 "],
})
 
print(df)

1) Fix text: `str.strip`, `str.lower`, `str.replace`

Normalize name/city

clean = df.copy()
clean["name"] = clean["name"].str.strip()
clean["city"] = clean["city"].str.strip().str.lower()
print(clean)

2) Convert string numbers to numeric: `to_numeric`

Remove commas and spaces first

Clean amount and convert

clean = df.copy()
clean["amount"] = clean["amount"].astype(str).str.replace(",", "", regex=False).str.strip()
clean["amount"] = pd.to_numeric(clean["amount"], errors="coerce")
 
print(clean)
print(clean.dtypes)

errors="coerce" turns invalid values into NaN (so you can handle them with dropna/fillna).

3) Change column types safely: `astype`

astype example

clean = clean.astype({"amount": "int64"})
print(clean.dtypes)

If conversion fails, Pandas throws an error. Prefer to_numeric(..., errors="coerce") for messy data.

4) Remove duplicates: `drop_duplicates`

Exact duplicates

Drop exact duplicates

dedup = clean.drop_duplicates()
print(dedup)

Duplicates based on a subset of columns

Duplicates by normalized name+city

dedup = clean.drop_duplicates(subset=["name", "city"], keep="first")
print(dedup)

Tip: validate your cleaning

After cleaning, quickly check:

df.dtypes
df.isna().sum()
df["col"].value_counts() for categories

Cleaning is not a single step. It’s a loop: inspect → clean → validate.

Visualize it

diagram The cleaning loop mermaid

  flowchart LR
  A["Inspect (dtypes, head)"] --> B["Clean (strip, replace, astype)"]
  B --> C["Validate (isna, value_counts)"]
  C -->|"still messy"| A
  C -->|"looks good"| D["Ready for analysis"]

Cleaning data is iterative: inspect, clean, validate, and repeat until the data is trustworthy.

🧪 Try It Yourself

Exercise 1 – Strip Whitespace From Text

Exercise 2 – Convert Text Numbers Safely

Exercise 3 – Drop Duplicate Rows

Text cleanup is worth its own deep dive — continue to Text and String Methods (str accessor and regex) to handle messier text patterns with the .str accessor and regex.

pch.coffeeTagline

pch.coffeeCta

pch.feedbackHeading

pch.feedbackSubheading