Data Type Conversion and Validation

Why dtype conversion matters

Many datasets arrive with wrong dtypes:

numbers stored as strings
dates stored as strings
categories stored inconsistently

If dtypes are wrong, your stats and charts can be wrong.

Convert to numeric safely

to_numeric

import pandas as pd
 
df = pd.DataFrame({"amount": ["1,200", "500", "oops", " 700 "]})
 
df["amount"] = df["amount"].astype(str).str.replace(",", "", regex=False).str.strip()
df["amount"] = pd.to_numeric(df["amount"], errors="coerce")
 
print(df)
print(df.isna().sum())

to_numeric

import pandas as pd
 
df = pd.DataFrame({"amount": ["1,200", "500", "oops", " 700 "]})
 
df["amount"] = df["amount"].astype(str).str.replace(",", "", regex=False).str.strip()
df["amount"] = pd.to_numeric(df["amount"], errors="coerce")
 
print(df)
print(df.isna().sum())

Convert to datetime

to_datetime

import pandas as pd
 
df = pd.DataFrame({"date": ["2025-01-01", "2025/01/02", "invalid"]})
 
df["date"] = pd.to_datetime(df["date"], errors="coerce")
print(df)

to_datetime

import pandas as pd
 
df = pd.DataFrame({"date": ["2025-01-01", "2025/01/02", "invalid"]})
 
df["date"] = pd.to_datetime(df["date"], errors="coerce")
print(df)

Validate assumptions

Typical checks:

ID columns contain no duplicates
numeric columns are non-negative
date columns are within expected range

Validation examples

# no duplicate ids
# assert df["id"].is_unique
 
# non-negative values
# assert (df["amount"] >= 0).all()
 
# date range
# assert df["date"].min() >= pd.Timestamp("2020-01-01")

Validation examples

# no duplicate ids
# assert df["id"].is_unique
 
# non-negative values
# assert (df["amount"] >= 0).all()
 
# date range
# assert df["date"].min() >= pd.Timestamp("2020-01-01")

Tip

Convert + validate early. It prevents subtle bugs later.

Visualize it

Each column starts life as text (or a mixed bag of types) and needs to be routed to the right converter before it’s safe to use.

diagram Dtype conversion flow mermaid

  flowchart LR
  A["Raw column (strings)"] --> B{"What kind of value?"}
  B -->|"numbers"| C["pd.to_numeric(errors='coerce')"]
  B -->|"dates"| D["pd.to_datetime(errors='coerce')"]
  B -->|"repeated labels"| E["astype('category')"]
  C --> F["Validated, typed column"]
  D --> F
  E --> F

Raw string columns are routed to a numeric, datetime, or category converter, with unparseable values becoming NA.

🧪 Try It Yourself

Exercise 1 – Convert text to numbers safely

Exercise 2 – Convert a column to category

Exercise 3 – Use a nullable integer dtype

With clean dtypes in hand, you’re ready to hunt for extreme values — see Outlier Detection (IQR Method) and Handling Outliers.

If this helped you, consider buying me a coffee ☕

Buy me a coffee