What you’ll learn

• three ways to handle missing values: drop rows, drop columns, or impute
• dropna()dropna(), drop()drop(), and fillna()fillna() in pandas
• SimpleImputerSimpleImputer from scikit-learn, and why it’s preferable to manual fillnafillna
• why you must fit the imputer on the training set only

Separate predictors from labels first

Before cleaning, split the training set into predictors and labels — you don’t necessarily want to apply the same transformations to both:

housing = strat_train_set.drop("median_house_value", axis=1)
housing_labels = strat_train_set["median_house_value"].copy()

housing = strat_train_set.drop("median_house_value", axis=1)
housing_labels = strat_train_set["median_house_value"].copy()

Three ways to handle missing values

Most Machine Learning algorithms can’t handle missing values at all. Earlier, housing.info()housing.info() showed that total_bedroomstotal_bedrooms has fewer non-null values than the rest of the dataset — you have three options:

# Option 1: get rid of the rows with missing values for this attribute
housing.dropna(subset=["total_bedrooms"])
 
# Option 2: get rid of the whole attribute
housing.drop("total_bedrooms", axis=1)
 
# Option 3: fill missing values with some value (zero, mean, median, ...)
median = housing["total_bedrooms"].median()
housing["total_bedrooms"].fillna(median, inplace=True)

# Option 1: get rid of the rows with missing values for this attribute
housing.dropna(subset=["total_bedrooms"])
 
# Option 2: get rid of the whole attribute
housing.drop("total_bedrooms", axis=1)
 
# Option 3: fill missing values with some value (zero, mean, median, ...)
median = housing["total_bedrooms"].median()
housing["total_bedrooms"].fillna(median, inplace=True)

If you pick option 3, compute the median from the training set only, and save that value — you’ll need it again to fill gaps in the test set later, and eventually in any new data the live system sees.

  flowchart TD
  A["Column has missing values"] --> B{"Pick a strategy"}
  B --> C["dropna: drop rows"]
  B --> D["drop: drop the column"]
  B --> E["fillna / SimpleImputer: impute a value"]
  E --> F["Fit imputer on TRAIN only"]
  F --> G["transform() train, test, and future data"]

SimpleImputer: the scikit-learn way

Filling values manually works, but doesn’t scale to many columns and doesn’t fit cleanly into a pipeline. SimpleImputerSimpleImputer handles the same idea in a reusable, scikit-learn-native way:

from sklearn.impute import SimpleImputer
 
imputer = SimpleImputer(strategy="median")
 
# median only makes sense for numeric columns — drop the one text column first
housing_num = housing.drop("ocean_proximity", axis=1)
 
imputer.fit(housing_num)
print(imputer.statistics_)
print(housing_num.median().values)

from sklearn.impute import SimpleImputer
 
imputer = SimpleImputer(strategy="median")
 
# median only makes sense for numeric columns — drop the one text column first
housing_num = housing.drop("ocean_proximity", axis=1)
 
imputer.fit(housing_num)
print(imputer.statistics_)
print(housing_num.median().values)

imputer.fit()imputer.fit() computes the median of every numeric column and stores it in statistics_statistics_ — even columns with no missing values get a stored median, since you can’t guarantee future data won’t have gaps elsewhere.

import pandas as pd
 
X = imputer.transform(housing_num)
 
housing_tr = pd.DataFrame(X, columns=housing_num.columns, index=housing_num.index)
print(housing_tr.isna().sum().sum())  # 0 — no missing values left

import pandas as pd
 
X = imputer.transform(housing_num)
 
housing_tr = pd.DataFrame(X, columns=housing_num.columns, index=housing_num.index)
print(housing_tr.isna().sum().sum())  # 0 — no missing values left

transform()transform() returns a plain NumPy array; wrapping it back in a DataFrameDataFrame (same columns, same index) makes it easy to keep working with pandas afterward.

Estimators, transformers, predictors

SimpleImputerSimpleImputer is a good first example of scikit-learn’s consistent API:

• Estimator — anything with a fit()fit() method that learns parameters from data (the imputer learning medians).
• Transformer — anything with a transform()transform() method that also transforms data (the imputer filling gaps). Most transformers also expose fit_transform()fit_transform() as a shortcut.
• Predictor — anything with a predict()predict() method (a trained model). You’ll meet this side of the API in later phases.

🧪 Try It Yourself

Exercise 1 – Drop rows with missing values

Exercise 2 – Fill missing values with the median

Exercise 3 – Use SimpleImputer

Next

Continue to Handling Text & Categorical Attributes — the ocean_proximityocean_proximity column is text, and most ML algorithms need numbers.