Data Type Conversion and Validation

Why dtype conversion matters

Many datasets arrive with wrong dtypes:

• numbers stored as strings
• dates stored as strings
• categories stored inconsistently

If dtypes are wrong, your stats and charts can be wrong.

Convert to numeric safely

to_numeric

import pandas as pd
 
df = pd.DataFrame({"amount": ["1,200", "500", "oops", " 700 "]})
 
df["amount"] = df["amount"].astype(str).str.replace(",", "", regex=False).str.strip()
df["amount"] = pd.to_numeric(df["amount"], errors="coerce")
 
print(df)
print(df.isna().sum())

Convert to datetime

to_datetime

import pandas as pd
 
df = pd.DataFrame({"date": ["2025-01-01", "2025/01/02", "invalid"]})
 
df["date"] = pd.to_datetime(df["date"], errors="coerce")
print(df)

Categories

category dtype

import pandas as pd
 
df = pd.DataFrame({"city": ["Pune", "Delhi", "Pune"]})
 
df["city"] = df["city"].astype("category")
print(df.dtypes)

Validate assumptions

Typical checks:

• ID columns contain no duplicates
• numeric columns are non-negative
• date columns are within expected range

Validation examples

# no duplicate ids
# assert df["id"].is_unique
 
# non-negative values
# assert (df["amount"] >= 0).all()
 
# date range
# assert df["date"].min() >= pd.Timestamp("2020-01-01")

Tip

Convert + validate early. It prevents subtle bugs later.

Visualize it

Each column starts life as text (or a mixed bag of types) and needs to be routed to the right converter before it’s safe to use.

diagram Dtype conversion flow mermaid

  flowchart LR
  A["Raw column (strings)"] --> B{"What kind of value?"}
  B -->|"numbers"| C["pd.to_numeric(errors='coerce')"]
  B -->|"dates"| D["pd.to_datetime(errors='coerce')"]
  B -->|"repeated labels"| E["astype('category')"]
  C --> F["Validated, typed column"]
  D --> F
  E --> F

Raw string columns are routed to a numeric, datetime, or category converter, with unparseable values becoming NA.

From the book

McKinney explains that pandas historically had to fall back to float64 and NaN to represent missing integers, because plain NumPy integer arrays can’t hold “missing.” The nullable extension dtypes — "Int64", "string", "boolean" (capital letters matter!) — fix this: they keep the real type and use pd.NA instead of silently upcasting your whole column to floats.

🧪 Try It Yourself

Exercise 1 – Convert text to numbers safely

pch.exerciseTag pch.exerciseTitle

Exercise 2 – Convert a column to category

pch.exerciseTag pch.exerciseTitle

Exercise 3 – Use a nullable integer dtype

pch.exerciseTag pch.exerciseTitle

Next

With clean dtypes in hand, you’re ready to hunt for extreme values — see Outlier Detection (IQR Method) and Handling Outliers.