What you’ll learn

• why you explore a copy of the training set and nothing else
• the Pearson correlation formula, computed by hand on five points
• the two things correlation is blind to, and the plot that exposes both
• how to spot data-quality artefacts that a summary statistic hides completely
• three engineered ratios, two of which outrank every raw count in the dataset
• the difference between a feature that correlates and a feature that helps

Intuition

The test set is locked away, so now you can be as aggressive as you like — but on a copy, so that experiments do not damage the training data itself.

Exploration has one goal: find the structure a model can use, and the defects that will mislead it. Both of those live in plots more often than in statistics. describe()describe() cannot tell you that two columns are latitude and longitude; a scatter plot draws California in three seconds.

housing = strat_train_set.copy()      # never explore the original, never touch the test set

housing = strat_train_set.copy()      # never explore the original, never touch the test set

The math: Pearson correlation

The numerator is the covariance — how the two vary together. The denominator normalises it by both spreads, so $r$ is dimensionless and confined to $[-1, +1]$. Because it is scale-free, the correlation between income and price is unchanged whether price is in dollars or thousands.

$r$ measures exactly one thing: the strength of a linear relationship. That precision is what makes it useful and what makes it dangerous.

Worked example by hand

Five districts. Income $x$ in tens of thousands, value $y$ in hundreds of thousands.

A strong positive linear relationship. Note what the arithmetic did: it never looked at the shape of the relationship, only at whether $x$ and $y$ deviate from their means in the same direction.

And $r^2 = 0.80$. For a simple linear regression of $y$ on $x$, the squared correlation is exactly the $R^2$ from Metrics R-Squared. Two chapters, one number.

Geography for free

housing.plot(
    kind="scatter", x="longitude", y="latitude", alpha=0.4,
    s=housing["population"] / 100, label="population",
    c="median_house_value", cmap="jet", colorbar=True, figsize=(10, 7),
)

housing.plot(
    kind="scatter", x="longitude", y="latitude", alpha=0.4,
    s=housing["population"] / 100, label="population",
    c="median_house_value", cmap="jet", colorbar=True, figsize=(10, 7),
)

figureTwo unlabelled numeric columns, plotted against each othermatplotlib

Reading the plot

1. Price follows the coast. Density and coastal proximity together explain far more than either alone — which is why ocean_proximityocean_proximity, the text column, turns out to matter.
2. Two bright clusters are the Bay Area and greater Los Angeles. A distance_to_nearest_citydistance_to_nearest_city feature is visible here and available nowhere in the raw columns.
3. The interior is uniformly cheap. That is the INLANDINLAND category, and it becomes the second most important feature in the final model.
4. Point size shows population, and the largest circles sit in the bright clusters. Correlated predictors, which matters when interpreting coefficients later.

What correlation cannot see

corr = housing.select_dtypes("number").corr()
print(corr["median_house_value"].sort_values(ascending=False).round(4).to_string())
 
# median_house_value    1.0000
# median_income         0.6881
# total_rooms           0.1342
# housing_median_age    0.1056
# households            0.0658
# total_bedrooms        0.0497
# population           -0.0246
# longitude            -0.0460
# latitude             -0.1442

corr = housing.select_dtypes("number").corr()
print(corr["median_house_value"].sort_values(ascending=False).round(4).to_string())
 
# median_house_value    1.0000
# median_income         0.6881
# total_rooms           0.1342
# housing_median_age    0.1056
# households            0.0658
# total_bedrooms        0.0497
# population           -0.0246
# longitude            -0.0460
# latitude             -0.1442

figureThe full numeric correlation matrixmatplotlib

Two blind spots matter here:

It only sees straight lines. A perfect parabola $y = x^2$ over symmetric $x$ has $r = 0$. Correlation zero does not mean “no relationship”, it means “no linear relationship”.

It only sees pairs. Latitude scores −0.14 and longitude −0.05, both apparently useless. The map above shows they are jointly among the strongest signals in the dataset. No pairwise statistic can find that, and in the final model latitude and longitude are the third and fourth most important features.

figureThe strongest relationship, at full resolutionmatplotlib

Reading the plot

• The upward trend is genuine, and $r = 0.69$ describes it fairly.
• The line at $500{,}001$ is the cap: 965 districts.
• Three fainter lines near 450k, 350k and 280k are also artefacts. Nobody documented these; only the scatter reveals them.
• The cloud fans out as income rises — heteroscedasticity, which does not bias the fit but does make uniform confidence intervals wrong.

A model trained on this data will reproduce those horizontal lines, because they are in the labels. Removing the capped districts is the honest fix, and it changes what the model is for.

What to do with a correlation number

A coefficient on its own is never an instruction. The number tells you which plot to draw next, and the plot tells you what to do:

  flowchart TD
  A["r for one column against the target"] --> B{"Is r large in
absolute value?"}
  B -->|"yes, e.g. income 0.69"| C["Scatter it against the target"]
  C --> D{"Straight cloud,
no stripes or walls?"}
  D -->|"clean"| E["Keep the column.
r describes it fairly."]
  D -->|"horizontal lines,
hard cap"| F["Recording artefact.
Drop capped rows or
reframe the target."]
  D -->|"fans out as x grows"| G["Heteroscedastic.
Fit is fine; uniform
intervals are not."]
  B -->|"no, e.g. latitude -0.14"| H{"Could it matter
jointly with another column?"}
  H -->|"geography, dates,
counts that scale together"| I["Plot the pair, or plot it
coloured by the target"]
  I --> J["Engineer a ratio,
interaction or cluster feature,
then re-measure r"]
  H -->|"no reason to expect it"| K{"Curved relationship
in the scatter?"}
  K -->|"yes, parabola"| L["r is blind here.
Use mutual information
or Spearman."]
  K -->|"no"| M["Genuinely weak.
Leave it in and let
the model decide."]

Every branch that ends in an action starts with a plot. That is the whole argument for EDA: the coefficient is a routing decision, not a conclusion.

See it move

The cloud below rotates and tightens while $r$ is recomputed live. Watch what happens when the relationship becomes a parabola: the points are perfectly determined and $r$ collapses to zero.

Engineering features

Raw counts in this dataset are nearly meaningless: total_roomstotal_rooms mostly measures how large a district is, not how desirable. Ratios fix that by dividing out district size.

housing["rooms_per_household"] = housing["total_rooms"] / housing["households"]
housing["bedrooms_per_room"] = housing["total_bedrooms"] / housing["total_rooms"]
housing["population_per_household"] = housing["population"] / housing["households"]
 
corr = housing.select_dtypes("number").corr()
print(corr["median_house_value"].sort_values(ascending=False).round(4).to_string())
 
# median_house_value          1.0000
# median_income               0.6881
# rooms_per_household         0.1519
# total_rooms                 0.1342
# housing_median_age          0.1056
# households                  0.0658
# total_bedrooms              0.0497
# population_per_household   -0.0237
# population                 -0.0246
# longitude                  -0.0460
# latitude                   -0.1442
# bedrooms_per_room          -0.2559

housing["rooms_per_household"] = housing["total_rooms"] / housing["households"]
housing["bedrooms_per_room"] = housing["total_bedrooms"] / housing["total_rooms"]
housing["population_per_household"] = housing["population"] / housing["households"]
 
corr = housing.select_dtypes("number").corr()
print(corr["median_house_value"].sort_values(ascending=False).round(4).to_string())
 
# median_house_value          1.0000
# median_income               0.6881
# rooms_per_household         0.1519
# total_rooms                 0.1342
# housing_median_age          0.1056
# households                  0.0658
# total_bedrooms              0.0497
# population_per_household   -0.0237
# population                 -0.0246
# longitude                  -0.0460
# latitude                   -0.1442
# bedrooms_per_room          -0.2559

figureThree new columns, from divisionmatplotlib

bedrooms_per_roombedrooms_per_room is the win. A low ratio means larger, more expensive homes — the raw total_bedroomstotal_bedrooms correlation of 0.05 was measuring district size and hiding this entirely. Two columns that individually say almost nothing produce, on division, the second-strongest signal in the dataset.

This is what feature engineering is: not more data, but the same data in a form that exposes the relationship.

Pitfalls

🧪 Try It Yourself

Exercise 1 – Correlation by hand

Exercise 2 – Sort correlations against the target

Exercise 3 – Engineer the winning ratio

Exercise 4 – Correlation is blind to curves

Exercise 5 – Find the recording artefacts

Recap

• Explore a copy of the training set. The test set stays closed.
• $r = \operatorname{Cov}(x,y) / (\sigma_x\sigma_y)$, dimensionless and bounded by $\pm 1$; worked by hand it gives 0.8944, and $r^2 = 0.80$ is the simple-regression $R^2$.
• Correlation sees only straight lines and only pairs. Latitude and longitude prove both limitations at once.
• Plotting two unlabelled columns drew a map, and the map showed that price follows the coast.
• The income scatter exposes a hard cap at $500,001 and three fainter recording artefacts.
• Three ratios: bedrooms_per_roombedrooms_per_room reaches −0.256, second only to income and stronger than every raw count.

Exercise 6 – The coefficient that sees curves, and the one that does not

Next

Continue to Data Cleaning & Handling Missing Values — deal with the 207 holes in total_bedroomstotal_bedrooms, and measure whether the choice of strategy actually changes anything.