Why “lifecycle” matters

ML isn’t just training a model.

A model that performs well in a notebook can fail in production because:

• the data distribution changes (drift)
• data quality issues appear
• latency constraints exist
• labels arrive late or are noisy

The lifecycle stages

  flowchart TD
  A[1. Problem Definition] --> B[2. Data Collection]
  B --> C[3. Data Cleaning & Preprocessing]
  C --> D[4. Feature Engineering]
  D --> E[5. Train Model]
  E --> F[6. Evaluate & Validate]
  F --> G[7. Deploy]
  G --> H[8. Monitor & Improve]
  H --> C

1) Problem definition

Decide:

• what is the target?
• what does success mean (metric + threshold)?
• what constraints exist (latency, cost, explainability)?

2) Data collection

Good data beats fancy models.

Typical sources:

• databases (SQL)
• CSV exports
• logs
• APIs

3) Cleaning & preprocessing

Examples:

• missing values
• outliers
• inconsistent categories
• duplicates

4) Feature engineering

Transform raw data into useful signals.

Example: from timestamps create:

• day-of-week
• hour-of-day

5) Training

Fit parameters of your chosen algorithm on training data.

6) Evaluation & validation

Use:

• validation sets and cross-validation
• metrics aligned with the business goal

Watch out for:

• data leakage
• overfitting

7) Deployment

Common forms:

• batch predictions (daily scoring job)
• real-time API
• embedded model (mobile/edge)

8) Monitoring

Monitor:

• input drift (feature distribution changes)
• prediction drift
• performance decay

Key takeaway

ML work is iterative.

You’ll move back and forth between data, features, and training until the model is good enough—and then keep iterating after deployment.

Main challenges: bad data

Since your only real levers are the algorithm and the data, most failures trace back to one of these:

• Insufficient training data — even simple problems often need thousands of examples; complex ones (vision, speech) may need millions. Famously, a 2001 Microsoft paper found that very different algorithms performed almost identically well once given enough data — sometimes data matters more than the algorithm.
• Nonrepresentative training data — if your sample is missing whole segments of the population, the model will not generalize to them. This is called sampling bias (the classic example: the 1936 Literary Digest poll predicted the wrong US president because it polled wealthier, telephone-owning households).
• Poor-quality data — errors, outliers, and noise make patterns harder to detect. Most data scientists spend a large share of their time just cleaning data.
• Irrelevant features — garbage in, garbage out. This is why feature engineering (selection, extraction, and creating new features) is so central to a project’s success.

Main challenges: bad algorithms (overfitting & underfitting)

• Overfitting — the model performs great on training data but fails to generalize, because it latched onto noise instead of the real pattern (like a high-degree polynomial that snakes through every training point). Fixes: simplify the model, gather more data, or reduce noise/outliers.
• Underfitting — the model is too simple to capture the underlying structure (like a straight line through a curved relationship), so it performs poorly even on the training data. Fixes: use a more powerful model, engineer better features, or relax any constraints on the model.

Testing and validation

The only real way to know if a model generalizes is to try it on data it has never seen:

• Training set — used to fit the model’s parameters.
• Test set — held out, used once at the end to estimate the generalization error (also called out-of-sample error). A common split is 80/20, adjusted for dataset size.
• Validation set — held out from training to compare candidate models and tune hyperparameters, so you don’t repeatedly “peek” at the test set (which would make your model overfit the test set itself). Averaging over several validation splits is called cross-validation.

Data mismatch: the train-dev set

Sometimes you can gather plenty of training data, but it doesn’t perfectly match what your model will see in production. Say you’re building a mobile app that identifies flower species from a photo. You can scrape millions of flower pictures off the web, but only 10,000 photos actually look like what the app’s camera produces (different lighting, angle, blur). If you train on the web photos and then get a disappointing score on your validation set, you can’t tell whether that’s overfitting or just a mismatch between the web pictures and real app pictures.

Andrew Ng’s fix: carve out a slice of the training data — never used during training — called the train-dev set.

• Train the model on the rest of the training set only.
• Then evaluate it on the train-dev set:
  • performs well → the model isn’t overfitting; a low score on the real validation set must be caused by the data mismatch instead.
  • performs poorly → the model overfit the training set — simplify it, regularize it, or gather more/cleaner training data.

Only the validation set and test set should be built exclusively from data that is representative of production (here, real photos taken with the app) — that’s the whole reason they exist: to tell you how the model will really perform once it ships.

No Free Lunch theorem

Every model makes assumptions about the data (a linear model assumes the data is roughly linear, for instance). David Wolpert’s 1996 No Free Lunch (NFL) theorem shows that if you make no assumptions at all, there’s no reason to prefer one model over another — no single model is guaranteed to be best for every dataset. In practice, you make reasonable assumptions about your problem and evaluate a handful of reasonable models rather than every model that exists.

Visualize it

ML is a cycle, not a one-shot task — the loop keeps running long after the first deployment.

  flowchart LR
  A[Collect data] --> B[Clean & prepare]
  B --> C[Train model]
  C --> D[Evaluate]
  D --> E[Deploy]
  E --> F[Monitor]
  F -->|retrain loop| A

🧪 Try It Yourself

Exercise 1 – Train-Test Split

Exercise 2 – Cross-Validation Instead of One Split

Exercise 3 – Spot Overfitting with Train vs Test Score