What you’ll learn

• why you explore a copy of the training set only, never the test set
• visualizing geographical data with scatterplots
• computing and reading a correlation matrix with corr()corr()
• the pandas scatter_matrix()scatter_matrix() for spotting relationships at a glance
• engineering new features by combining existing attributes

Explore a copy of the training set

With the test set safely locked away, you’re free to dig into the training data. If the training set is huge, you might sample a smaller exploration set to keep things fast — here it’s small enough to just work on a copy directly:

housing = strat_train_set.copy()

housing = strat_train_set.copy()

Visualizing geographical data

Since the dataset has longitudelongitude/latitudelatitude, a scatterplot is the obvious first move:

housing.plot(kind="scatter", x="longitude", y="latitude", alpha=0.1)

housing.plot(kind="scatter", x="longitude", y="latitude", alpha=0.1)

Setting alpha=0.1alpha=0.1 is the trick that makes this useful — with thousands of overlapping points, a solid dot per district just looks like a blob of California. Low alpha lets dense areas show up darker, revealing the Bay Area, LA, San Diego, and a line of density through the Central Valley.

You can pack even more into one plot: circle radius for population, color for price.

import matplotlib.pyplot as plt
 
housing.plot(
    kind="scatter", x="longitude", y="latitude", alpha=0.4,
    s=housing["population"] / 100, label="population", figsize=(10, 7),
    c="median_house_value", cmap=plt.get_cmap("jet"), colorbar=True,
)
plt.legend()
plt.show()

import matplotlib.pyplot as plt
 
housing.plot(
    kind="scatter", x="longitude", y="latitude", alpha=0.4,
    s=housing["population"] / 100, label="population", figsize=(10, 7),
    c="median_house_value", cmap=plt.get_cmap("jet"), colorbar=True,
)
plt.legend()
plt.show()

This single chart tells a strong story: prices correlate with location (closer to the ocean tends to cost more) and with population density.

Looking for correlations

Since this dataset is small, you can compute the standard correlation coefficient (Pearson’s r) between every pair of numeric attributes with one call:

corr_matrix = housing.corr(numeric_only=True)
print(corr_matrix["median_house_value"].sort_values(ascending=False))

corr_matrix = housing.corr(numeric_only=True)
print(corr_matrix["median_house_value"].sort_values(ascending=False))

median_house_value    1.000000
median_income         0.687170
total_rooms           0.135231
housing_median_age    0.114220
households            0.064702
total_bedrooms        0.047865
population           -0.026699
longitude             -0.047279
latitude              -0.142826

median_house_value    1.000000
median_income         0.687170
total_rooms           0.135231
housing_median_age    0.114220
households            0.064702
total_bedrooms        0.047865
population           -0.026699
longitude             -0.047279
latitude              -0.142826

The coefficient ranges from -1 to 1. Close to 1 means a strong positive correlation (price rises with income); close to -1 means a strong negative correlation (price drops slightly the further north you go); close to 0 means no linear relationship — though it can still miss strong non-linear patterns entirely, so don’t treat 0 as “unrelated.”

pandas.plotting.scatter_matrix()pandas.plotting.scatter_matrix() plots every attribute against every other one, which gets unwieldy fast — focus it on the attributes most correlated with your target:

from pandas.plotting import scatter_matrix
 
attributes = ["median_house_value", "median_income", "total_rooms", "housing_median_age"]
scatter_matrix(housing[attributes], figsize=(12, 8))

from pandas.plotting import scatter_matrix
 
attributes = ["median_house_value", "median_income", "total_rooms", "housing_median_age"]
scatter_matrix(housing[attributes], figsize=(12, 8))

Zooming in on the single strongest pair confirms the relationship and reveals data quirks worth knowing about — like a hard price cap at $500,000 showing up as a suspicious flat line:

housing.plot(kind="scatter", x="median_income", y="median_house_value", alpha=0.1)

housing.plot(kind="scatter", x="median_income", y="median_house_value", alpha=0.1)

  flowchart LR
  A["Raw columns
(total_rooms, households, ...)"] --> B["corr() with target"]
  B --> C["Spot weak but combinable signals"]
  C --> D["Engineer ratios
(rooms_per_household, ...)"]
  D --> E["Recheck corr() -
often stronger than the originals"]

Experimenting with attribute combinations

total_roomstotal_rooms alone isn’t very informative without knowing how many households share them. Ratios are often far more predictive than the raw counts:

housing["rooms_per_household"] = housing["total_rooms"] / housing["households"]
housing["bedrooms_per_room"] = housing["total_bedrooms"] / housing["total_rooms"]
housing["population_per_household"] = housing["population"] / housing["households"]
 
corr_matrix = housing.corr(numeric_only=True)
print(corr_matrix["median_house_value"].sort_values(ascending=False))

housing["rooms_per_household"] = housing["total_rooms"] / housing["households"]
housing["bedrooms_per_room"] = housing["total_bedrooms"] / housing["total_rooms"]
housing["population_per_household"] = housing["population"] / housing["households"]
 
corr_matrix = housing.corr(numeric_only=True)
print(corr_matrix["median_house_value"].sort_values(ascending=False))

In the book’s run, bedrooms_per_roombedrooms_per_room correlates with price far more strongly (-0.26) than either total_roomstotal_rooms or total_bedroomstotal_bedrooms alone — districts with a lower bedroom-to-room ratio tend to be more expensive. rooms_per_householdrooms_per_household also beats the raw total_roomstotal_rooms count. This kind of feature engineering, guided by correlation analysis, often does more for model quality than swapping algorithms.

🧪 Try It Yourself

Exercise 1 – Compute a correlation matrix

Exercise 2 – Sort correlations against the target

Exercise 3 – Engineer a ratio feature

Next

Continue to Data Cleaning & Handling Missing Values — before feeding this data to any algorithm, you need to deal with the gaps you spotted in total_bedroomstotal_bedrooms.