Data Type Conversion and Validation

Why dtype conversion matters

Many datasets arrive with wrong dtypes:

• numbers stored as strings
• dates stored as strings
• categories stored inconsistently

If dtypes are wrong, your stats and charts can be wrong.

Convert to numeric safely

import pandas as pd
 
df = pd.DataFrame({"amount": ["1,200", "500", "oops", " 700 "]})
 
df["amount"] = df["amount"].astype(str).str.replace(",", "", regex=False).str.strip()
df["amount"] = pd.to_numeric(df["amount"], errors="coerce")
 
print(df)
print(df.isna().sum())

Convert to datetime

import pandas as pd
 
df = pd.DataFrame({"date": ["2025-01-01", "2025/01/02", "invalid"]})
 
df["date"] = pd.to_datetime(df["date"], errors="coerce")
print(df)

Categories

import pandas as pd
 
df = pd.DataFrame({"city": ["Pune", "Delhi", "Pune"]})
 
df["city"] = df["city"].astype("category")
print(df.dtypes)

Validate assumptions

Typical checks:

• ID columns contain no duplicates
• numeric columns are non-negative
• date columns are within expected range

# no duplicate ids
# assert df["id"].is_unique
 
# non-negative values
# assert (df["amount"] >= 0).all()
 
# date range
# assert df["date"].min() >= pd.Timestamp("2020-01-01")

Tip

Convert + validate early. It prevents subtle bugs later.

Visualize it

Each column starts life as text (or a mixed bag of types) and needs to be routed to the right converter before it's safe to use.

  flowchart LR
  A["Raw column (strings)"] --> B{"What kind of value?"}
  B -->|"numbers"| C["pd.to_numeric(errors='coerce')"]
  B -->|"dates"| D["pd.to_datetime(errors='coerce')"]
  B -->|"repeated labels"| E["astype('category')"]
  C --> F["Validated, typed column"]
  D --> F
  E --> F

🧪 Try It Yourself

Exercise 1 – Convert text to numbers safely

Press Run to see what your code prints.

Exercise 2 – Convert a column to category

Press Run to see what your code prints.

Exercise 3 – Use a nullable integer dtype

Press Run to see what your code prints.

Check your understanding

Next

With clean dtypes in hand, you're ready to hunt for extreme values — see Outlier Detection (IQR Method) and Handling Outliers.