The ML Pipeline - Automating the Workflow

Every technique in this phase — cross-validation, grid search, randomized search — only gives trustworthy numbers if preprocessing is redone correctly on every fold. Scale the entire dataset before splitting, and you’ve leaked information about the validation fold into training; do it by hand every time and you will eventually get it wrong. PipelinePipeline and ColumnTransformerColumnTransformer turn “preprocess, then model” into one object that gets it right automatically, every single time.

Why pipelines matter

Pipelines prevent a common ML failure:

preprocessing done differently in train vs test vs production

They also prevent leakage by ensuring:

preprocessors are fit only on training folds during CV

The pattern

diagram Diagram mermaid

  flowchart LR
  X[Raw X] --> P["Preprocess (fit on train)"] --> M[Model] --> Y[Predictions]

Pipeline components

PipelinePipeline: chain steps
ColumnTransformerColumnTransformer: apply different transforms to numeric vs categorical columns

Example: end-to-end tabular pipeline

Preprocess + model pipeline

from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
 
numeric_features = ["age", "income"]
categorical_features = ["city", "plan"]
 
numeric_pipe = Pipeline(
    steps=[
        ("imputer", SimpleImputer(strategy="median")),
        ("scaler", StandardScaler()),
    ]
)
 
categorical_pipe = Pipeline(
    steps=[
        ("imputer", SimpleImputer(strategy="most_frequent")),
        ("onehot", OneHotEncoder(handle_unknown="ignore")),
    ]
)
 
preprocess = ColumnTransformer(
    transformers=[
        ("num", numeric_pipe, numeric_features),
        ("cat", categorical_pipe, categorical_features),
    ]
)
 
model = Pipeline(
    steps=[
        ("preprocess", preprocess),
        ("clf", LogisticRegression(max_iter=1000)),
    ]
)

Preprocess + model pipeline

from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
 
numeric_features = ["age", "income"]
categorical_features = ["city", "plan"]
 
numeric_pipe = Pipeline(
    steps=[
        ("imputer", SimpleImputer(strategy="median")),
        ("scaler", StandardScaler()),
    ]
)
 
categorical_pipe = Pipeline(
    steps=[
        ("imputer", SimpleImputer(strategy="most_frequent")),
        ("onehot", OneHotEncoder(handle_unknown="ignore")),
    ]
)
 
preprocess = ColumnTransformer(
    transformers=[
        ("num", numeric_pipe, numeric_features),
        ("cat", categorical_pipe, categorical_features),
    ]
)
 
model = Pipeline(
    steps=[
        ("preprocess", preprocess),
        ("clf", LogisticRegression(max_iter=1000)),
    ]
)

Pipelines + hyperparameter tuning

You can tune model params inside the pipeline using the step name:

clf__Cclf__C
clf__penaltyclf__penalty

Grid search inside a pipeline

from sklearn.model_selection import GridSearchCV
 
param_grid = {
    "clf__C": [0.1, 1, 10],
    "clf__penalty": ["l2"],
}
 
grid = GridSearchCV(model, param_grid, cv=5, scoring="accuracy")
grid.fit(X_train, y_train)
print("best params:", grid.best_params_)

Grid search inside a pipeline

from sklearn.model_selection import GridSearchCV
 
param_grid = {
    "clf__C": [0.1, 1, 10],
    "clf__penalty": ["l2"],
}
 
grid = GridSearchCV(model, param_grid, cv=5, scoring="accuracy")
grid.fit(X_train, y_train)
print("best params:", grid.best_params_)

Mini-checkpoint

If you’re still scaling outside a pipeline:

you’re one step away from leakage.

🧪 Try It Yourself

Exercise 1 – Chain Steps with Pipeline

Exercise 2 – Route Columns with ColumnTransformer

Exercise 3 – Tune a Model Nested Inside a Pipeline

Continue to Saving and Loading Models (Pickle, Joblib) in Phase 8 — persist this tuned pipeline so you don’t have to retrain it every time.

If this helped you, consider buying me a coffee ☕

Buy me a coffee