Why this phase exists

Every Machine Learning course jumps to model.fit()model.fit() fast. Real projects don’t work that way. Before any algorithm ever sees your data, someone has to frame the problem, pull a representative test set aside, poke at the data to understand it, patch up the messy parts, and turn raw columns into numbers a model can actually learn from.

This phase follows Chapter 2 of Hands-On Machine Learning (Géron) — the “end-to-end project” chapter — using a single running example: predicting median house prices for California districts from 1990 census data. You will:

• frame the ML problem and pick a performance measure (RMSE / MAE)
• split off a test set correctly, so you never fool yourself with data snooping
• explore the training data and look for correlations and useful feature combinations
• clean missing values, encode text/categorical columns, and scale numeric features
• wrap all of that into a single reusable PipelinePipeline / ColumnTransformerColumnTransformer

By the end, you’ll have a full_pipeline.fit_transform(...)full_pipeline.fit_transform(...) that turns a raw CSV into a matrix ready for model.fit()model.fit() — the boring-but-essential 80% of a real ML project.

The end-to-end pipeline

  flowchart TD
  A["Frame the problem"] --> B["Get the data
(pandas.read_csv)"]
  B --> C["Create a test set
(stratified split)"]
  C --> D["Explore & visualize
(training set only)"]
  D --> E["Clean missing values
(SimpleImputer)"]
  E --> F["Encode text/categorical
(OneHotEncoder)"]
  F --> G["Scale numeric features
(StandardScaler)"]
  G --> H["Pipeline + ColumnTransformer"]
  H --> I["Model-ready matrix"]

What’s in this phase

1. Framing an ML Problem & Getting the Data — is it supervised, regression, batch learning? Load the housing CSV with pandas and take a first look.
2. Creating a Test Set (Avoiding Data Snooping) — random vs. stratified sampling, and why looking at your test set too early quietly ruins your evaluation.
3. Exploratory Data Analysis & Correlations — scatterplots, corr()corr(), and combining attributes into more informative features.
4. Data Cleaning & Handling Missing Values — dropnadropna, dropdrop, and SimpleImputerSimpleImputer.
5. Handling Text & Categorical Attributes — OrdinalEncoderOrdinalEncoder vs. OneHotEncoderOneHotEncoder.
6. Feature Scaling (Normalization & Standardization) — MinMaxScalerMinMaxScaler vs. StandardScalerStandardScaler, and why you only ever fit on the training set.
7. Transformation Pipelines & Custom Transformers — gluing everything into one PipelinePipeline and ColumnTransformerColumnTransformer, plus writing your own scikit-learn-compatible transformer.

Next

Start with Framing an ML Problem & Getting the Data to see how a vague business request turns into a concrete regression task with a measurable target.