Handling Missing Data (isna, fillna, dropna)

Why missing data is normal

Missing values happen because of:

Optional form fields (e.g., phone number)
Data entry errors
Failed joins/merges
Incomplete logs

Pandas typically represents missing values as NaN (and sometimes None).

Setup example with missing values

Missing values example

import pandas as pd
import numpy as np
 
df = pd.DataFrame({
    "name": ["Asha", "Ravi", "Meera", None],
    "age": [23, np.nan, 26, 31],
    "city": ["Pune", "Delhi", None, "Pune"],
    "score": [88, 91, np.nan, 95],
})
 
print(df)

Detect missing values

`isna()` / `isnull()`

isna

print(df.isna())

Count missing per column

Missing counts

print(df.isna().sum())

Remove missing values: `dropna()`

Drop rows with any missing values

Drop rows with any NA

clean = df.dropna()
print(clean)

Drop rows where a specific column is missing

Drop rows where score is missing

clean = df.dropna(subset=["score"])
print(clean)

Fill missing values: `fillna()`

Fill with a constant

Fill missing city

filled = df.copy()
filled["city"] = filled["city"].fillna("Unknown")
print(filled)

Fill numeric missing values with mean/median

Fill numeric with median

filled = df.copy()
filled["score"] = filled["score"].fillna(filled["score"].median())
print(filled)

Forward fill / backward fill

Useful for time series or repeated categories.

Forward fill

filled = df.copy()
filled["city"] = filled["city"].ffill()
print(filled)

Important: be explicit

Before deciding how to handle missing data, ask:

Is missingness random or meaningful?
Should missing values be removed or imputed?
Will filling change the meaning of the data?

In analytics, documenting missing-data decisions is part of good practice.

Visualize it

diagram Two ways to handle missing values mermaid

  flowchart LR
  A["Raw data with NaN"] --> B["df.isna()"]
  B --> C{"Drop or fill?"}
  C -->|"dropna()"| D["Rows removed"]
  C -->|"fillna()"| E["Values imputed"]

After detecting missing values, you either drop the affected rows/columns or fill them in.

🧪 Try It Yourself

Exercise 1 – Detect Missing Values

Exercise 2 – Drop Rows With Any Missing Value

Exercise 3 – Fill Missing Values With the Mean

With missing values under control, move on to Cleaning Data (astype, duplicates, string cleaning) to fix types, duplicates, and messy text.

pch.coffeeTagline

pch.coffeeCta

pch.feedbackHeading

pch.feedbackSubheading