Why missing data is normal
Missing values happen because of:
- Optional form fields (e.g., phone number)
- Data entry errors
- Failed joins/merges
- Incomplete logs
Pandas typically represents missing values as NaN (and sometimes None).
Setup example with missing values
import pandas as pd
import numpy as np
df = pd.DataFrame({
"name": ["Asha", "Ravi", "Meera", None],
"age": [23, np.nan, 26, 31],
"city": ["Pune", "Delhi", None, "Pune"],
"score": [88, 91, np.nan, 95],
})
print(df)Detect missing values
isna() / isnull()
print(df.isna())Count missing per column
print(df.isna().sum())Remove missing values: dropna()
Drop rows with any missing values
clean = df.dropna()
print(clean)Drop rows where a specific column is missing
clean = df.dropna(subset=["score"])
print(clean)Fill missing values: fillna()
Fill with a constant
filled = df.copy()
filled["city"] = filled["city"].fillna("Unknown")
print(filled)Fill numeric missing values with mean/median
filled = df.copy()
filled["score"] = filled["score"].fillna(filled["score"].median())
print(filled)Forward fill / backward fill
Useful for time series or repeated categories.
filled = df.copy()
filled["city"] = filled["city"].ffill()
print(filled)Important: be explicit
Before deciding how to handle missing data, ask:
- Is missingness random or meaningful?
- Should missing values be removed or imputed?
- Will filling change the meaning of the data?
In analytics, documenting missing-data decisions is part of good practice.
Visualize it
flowchart LR
A["Raw data with NaN"] --> B["df.isna()"]
B --> C{"Drop or fill?"}
C -->|"dropna()"| D["Rows removed"]
C -->|"fillna()"| E["Values imputed"]
🧪 Try It Yourself
Exercise 1 – Detect Missing Values
Tab indents. Press Escape, then Tab, to leave the editor. Control plus Enter runs the code; Control plus Shift plus Enter submits it.
Press Run to see what your code prints.Exercise 2 – Drop Rows With Any Missing Value
Tab indents. Press Escape, then Tab, to leave the editor. Control plus Enter runs the code; Control plus Shift plus Enter submits it.
Press Run to see what your code prints.Exercise 3 – Fill Missing Values With the Mean
Tab indents. Press Escape, then Tab, to leave the editor. Control plus Enter runs the code; Control plus Shift plus Enter submits it.
Press Run to see what your code prints.Check your understanding
What does df.isna().sum() return?
isna() marks missing cells as True and sum() adds them up per column.
Show answer
A — The number of missing values in each column — isna() marks missing cells as True and sum() adds them up per column.
In the sample data, only Asha's row has no missing values. What does df.dropna() return?
dropna() removes every row with any missing value, and Ravi, Meera and the row with name None each have at least one.
Show answer
B — Just 1 row — dropna() removes every row with any missing value, and Ravi, Meera and the row with name None each have at least one.
What does df.dropna(subset=["score"]) do?
subset limits the check to the listed columns, so only Meera's row (score NaN) is removed.
Show answer
A — Drops rows where score is missing — subset limits the check to the listed columns, so only Meera's row (score NaN) is removed.
Why fill missing scores with the median instead of 0?
For scores 88, 91 and 95 the median is 91, a realistic fill. Filling with 0 would drag down averages and change the meaning of the data.
Show answer
A — The median keeps the typical level of the data, while 0 would distort it — For scores 88, 91 and 95 the median is 91, a realistic fill. Filling with 0 would drag down averages and change the meaning of the data.
Next
With missing values under control, move on to Cleaning Data (astype, duplicates, string cleaning) to fix types, duplicates, and messy text.