What you’ll learn

• why hand-running each preprocessing step is error-prone
• chaining steps with PipelinePipeline
• applying different pipelines to different columns with ColumnTransformerColumnTransformer
• writing a custom transformer with BaseEstimatorBaseEstimator and TransformerMixinTransformerMixin
• running the full pipeline end to end on raw data

The problem with doing it by hand

By now you’ve written separate code for imputing, encoding, and scaling — each correct, but easy to apply in the wrong order, forget on the test set, or duplicate across notebooks. PipelinePipeline chains a sequence of transformers (plus one final estimator) into a single object with the same fit()fit() / transform()transform() interface as any of its individual steps.

from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler
 
num_pipeline = Pipeline([
    ("imputer", SimpleImputer(strategy="median")),
    ("std_scaler", StandardScaler()),
])
 
housing_num_tr = num_pipeline.fit_transform(housing_num)

from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler
 
num_pipeline = Pipeline([
    ("imputer", SimpleImputer(strategy="median")),
    ("std_scaler", StandardScaler()),
])
 
housing_num_tr = num_pipeline.fit_transform(housing_num)

Each step is a (name, transformer)(name, transformer) pair. Every step except the last must implement fit_transform()fit_transform(); calling fit()fit() on the pipeline runs fit_transform()fit_transform() on each step in turn, feeding the output forward, until it reaches the final estimator. Names can be anything unique (just avoid double underscores — they’re reserved for hyperparameter tuning later).

Writing a custom transformer

Some transformations — like the rooms_per_householdrooms_per_household and bedrooms_per_roombedrooms_per_room combinations from earlier — aren’t built into scikit-learn. Because scikit-learn relies on duck typing rather than inheritance, you only need to implement fit()fit() (returning selfself), transform()transform(), and fit_transform()fit_transform() — and you get the last one for free by inheriting from TransformerMixinTransformerMixin. Inheriting BaseEstimatorBaseEstimator too (and avoiding *args*args/**kwargs**kwargs in __init____init__) gives you get_params()get_params() / set_params()set_params() for free, which matters later for automated hyperparameter search.

import numpy as np
from sklearn.base import BaseEstimator, TransformerMixin
 
rooms_ix, bedrooms_ix, population_ix, households_ix = 3, 4, 5, 6
 
class CombinedAttributesAdder(BaseEstimator, TransformerMixin):
    def __init__(self, add_bedrooms_per_room=True):
        self.add_bedrooms_per_room = add_bedrooms_per_room
 
    def fit(self, X, y=None):
        return self  # nothing to learn
 
    def transform(self, X):
        rooms_per_household = X[:, rooms_ix] / X[:, households_ix]
        population_per_household = X[:, population_ix] / X[:, households_ix]
        if self.add_bedrooms_per_room:
            bedrooms_per_room = X[:, bedrooms_ix] / X[:, rooms_ix]
            return np.c_[X, rooms_per_household, population_per_household, bedrooms_per_room]
        return np.c_[X, rooms_per_household, population_per_household]

import numpy as np
from sklearn.base import BaseEstimator, TransformerMixin
 
rooms_ix, bedrooms_ix, population_ix, households_ix = 3, 4, 5, 6
 
class CombinedAttributesAdder(BaseEstimator, TransformerMixin):
    def __init__(self, add_bedrooms_per_room=True):
        self.add_bedrooms_per_room = add_bedrooms_per_room
 
    def fit(self, X, y=None):
        return self  # nothing to learn
 
    def transform(self, X):
        rooms_per_household = X[:, rooms_ix] / X[:, households_ix]
        population_per_household = X[:, population_ix] / X[:, households_ix]
        if self.add_bedrooms_per_room:
            bedrooms_per_room = X[:, bedrooms_ix] / X[:, rooms_ix]
            return np.c_[X, rooms_per_household, population_per_household, bedrooms_per_room]
        return np.c_[X, rooms_per_household, population_per_household]

The add_bedrooms_per_roomadd_bedrooms_per_room flag is a hyperparameter with a sensible default — exactly the kind of on/off switch you’d want to try both ways during model tuning, without rewriting the transformer.

num_pipeline = Pipeline([
    ("imputer", SimpleImputer(strategy="median")),
    ("attribs_adder", CombinedAttributesAdder()),
    ("std_scaler", StandardScaler()),
])
 
housing_num_tr = num_pipeline.fit_transform(housing_num)

num_pipeline = Pipeline([
    ("imputer", SimpleImputer(strategy="median")),
    ("attribs_adder", CombinedAttributesAdder()),
    ("std_scaler", StandardScaler()),
])
 
housing_num_tr = num_pipeline.fit_transform(housing_num)

ColumnTransformer: numeric and categorical, together

So far, numeric and categorical columns have been handled with two separate code paths. ColumnTransformerColumnTransformer applies a different transformer to each named list of columns and concatenates the results — including a mix of dense and sparse output.

from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import OneHotEncoder
 
num_attribs = list(housing_num)
cat_attribs = ["ocean_proximity"]
 
full_pipeline = ColumnTransformer([
    ("num", num_pipeline, num_attribs),
    ("cat", OneHotEncoder(), cat_attribs),
])
 
housing_prepared = full_pipeline.fit_transform(housing)

from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import OneHotEncoder
 
num_attribs = list(housing_num)
cat_attribs = ["ocean_proximity"]
 
full_pipeline = ColumnTransformer([
    ("num", num_pipeline, num_attribs),
    ("cat", OneHotEncoder(), cat_attribs),
])
 
housing_prepared = full_pipeline.fit_transform(housing)

The constructor takes a list of (name, transformer, columns)(name, transformer, columns) tuples. Numeric columns flow through num_pipelinenum_pipeline (impute → engineer → scale); the categorical column flows through OneHotEncoderOneHotEncoder. ColumnTransformerColumnTransformer runs each transformer on its own subset of columns and stitches the outputs back together along the column axis — automatically deciding whether the result should be sparse or dense based on how much of it is actually non-zero.

  flowchart LR
  A["housing DataFrame"] --> B["ColumnTransformer"]
  B --> C["num_attribs -> num_pipeline
(impute, engineer, scale)"]
  B --> D["cat_attribs -> OneHotEncoder"]
  C --> E["Concatenate outputs"]
  D --> E
  E --> F["housing_prepared
(model-ready matrix)"]

Using the full pipeline

Once fit, full_pipelinefull_pipeline behaves like any other transformer — fit_transform()fit_transform() on the training set, transform()transform() (never fit_transform()fit_transform() again!) on the test set or any new data:

some_data = housing.iloc[:5]
some_data_prepared = full_pipeline.transform(some_data)
print(some_data_prepared.shape)

some_data = housing.iloc[:5]
some_data_prepared = full_pipeline.transform(some_data)
print(some_data_prepared.shape)

That single call reproduces every step this whole phase covered — imputing, engineered ratios, one-hot encoding, and scaling — consistently and in the right order, every time.

🧪 Try It Yourself

Exercise 1 – Chain steps with Pipeline

Exercise 2 – Write a custom transformer

Exercise 3 – Combine pipelines with ColumnTransformer

Next

With housing_preparedhousing_prepared ready — every column numeric, engineered, and scaled — you’re set up for Phase 3: Supervised Learning - Regression, where this same data is used to train and evaluate an actual regression model.