Handling Outliers

Outliers are not always errors

Outliers could be:

• True rare events (high-value orders)
• Measurement/unit problems
• Recording errors
• Different segment (VIP customers)

So the first step is always: investigate.

Common strategies

1) Investigate and correct (best)

• Check the source system
• Confirm units
• Validate against business rules

2) Remove outliers (use carefully)

Remove via IQR bounds

clean = df[(df["amount"] >= lower) & (df["amount"] <= upper)].copy()
print(clean)

Removing is risky when you report business totals.

3) Cap / winsorize

Capping keeps all rows but limits extreme values.

Cap values

df["amount_capped"] = df["amount"].clip(lower, upper)

4) Transform (e.g., log)

Useful when values span many orders of magnitude.

Log transform

import numpy as np
 
df["amount_log"] = np.log1p(df["amount"])  # log(1+x) to handle 0

Which strategy should you pick?

• Reporting metrics → investigate, maybe cap
• ML features → cap or transform often helps
• Fraud/anomaly detection → keep outliers (they may be the signal)

Always document

• detection rule
• chosen handling method
• expected impact

Visualize it

diagram Handling outliers decision mermaid

  flowchart LR
  A["Outlier detected"] --> B{"Why is it there?"}
  B -->|"data error"| C["Investigate & correct"]
  B -->|"true rare event"| D["Keep or flag"]
  B -->|"reporting / ML feature"| E["Cap (winsorize)"]
  B -->|"skewed magnitude"| F["Transform (log)"]

After detecting an outlier, the right response depends on why it exists and what the data will be used for.

Most values cluster tightly together; an outlier sits far away from that cluster. Capping doesn’t delete it — it pulls the value back to a boundary you choose.

sketch An outlier on a number line p5.js Pause Restart

Normal values cluster together; an outlier sits far away and can be pulled back to a cap boundary instead of deleted.

From the book

McKinney’s capping trick for a whole DataFrame at once: data[data.abs() > 3] = np.sign(data) * 3. np.sign(data) gives +1 or -1 for every value, so multiplying by 3 pulls every extreme value back to exactly +3 or -3 while keeping its original sign.

🧪 Try It Yourself

Exercise 1 – Cap values with clip

pch.exerciseTag pch.exerciseTitle

Exercise 2 – Cap using the sign trick

pch.exerciseTag pch.exerciseTitle

Exercise 3 – Transform with log1p

pch.exerciseTag pch.exerciseTitle

Next

Now that extreme values are under control, move on to Feature Scaling (MinMax vs Standard) so all your numeric features live on comparable scales.