Outliers are not always errors
Outliers could be:
- True rare events (high-value orders)
- Measurement/unit problems
- Recording errors
- Different segment (VIP customers)
So the first step is always: investigate.
Common strategies
1) Investigate and correct (best)
- Check the source system
- Confirm units
- Validate against business rules
2) Remove outliers (use carefully)
clean = df[(df["amount"] >= lower) & (df["amount"] <= upper)].copy()
print(clean)Removing is risky when you report business totals.
3) Cap / winsorize
Capping keeps all rows but limits extreme values.
df["amount_capped"] = df["amount"].clip(lower, upper)4) Transform (e.g., log)
Useful when values span many orders of magnitude.
import numpy as np
df["amount_log"] = np.log1p(df["amount"]) # log(1+x) to handle 0Which strategy should you pick?
- Reporting metrics → investigate, maybe cap
- ML features → cap or transform often helps
- Fraud/anomaly detection → keep outliers (they may be the signal)
Always document
- detection rule
- chosen handling method
- expected impact
Visualize it
flowchart LR
A["Outlier detected"] --> B{"Why is it there?"}
B -->|"data error"| C["Investigate & correct"]
B -->|"true rare event"| D["Keep or flag"]
B -->|"reporting / ML feature"| E["Cap (winsorize)"]
B -->|"skewed magnitude"| F["Transform (log)"]
Most values cluster tightly together; an outlier sits far away from that cluster. Capping doesn't delete it — it pulls the value back to a boundary you choose.
🧪 Try It Yourself
Exercise 1 – Cap values with clip
Tab indents. Press Escape, then Tab, to leave the editor. Control plus Enter runs the code; Control plus Shift plus Enter submits it.
Press Run to see what your code prints.Exercise 2 – Cap using the sign trick
Tab indents. Press Escape, then Tab, to leave the editor. Control plus Enter runs the code; Control plus Shift plus Enter submits it.
Press Run to see what your code prints.Exercise 3 – Transform with log1p
Tab indents. Press Escape, then Tab, to leave the editor. Control plus Enter runs the code; Control plus Shift plus Enter submits it.
Press Run to see what your code prints.Check your understanding
What is the first step the lesson recommends when you find an outlier?
Outliers may be rare true events, unit problems, recording errors or a different segment. Check the source, units and business rules first.
Show answer
A — Investigate it — Outliers may be rare true events, unit problems, recording errors or a different segment. Check the source, units and business rules first.
What does df["amount"].clip(lower, upper) do?
Capping (winsorizing) pulls extreme values back to a boundary you choose rather than removing them.
Show answer
A — Caps extreme values at the bounds while keeping every row — Capping (winsorizing) pulls extreme values back to a boundary you choose rather than removing them.
Why does the lesson use np.log1p instead of np.log?
log1p computes log(1 + x), so a value of 0 does not break the transform. Log transforms help when values span many orders of magnitude.
Show answer
A — log(1 + x) handles zeros — log1p computes log(1 + x), so a value of 0 does not break the transform. Log transforms help when values span many orders of magnitude.
For fraud or anomaly detection, what does the lesson say to do with outliers?
In fraud detection the unusual values are exactly what you are looking for.
Show answer
A — Keep them, since they may be the signal — In fraud detection the unusual values are exactly what you are looking for.
Next
Now that extreme values are under control, move on to Feature Scaling (MinMax vs Standard) so all your numeric features live on comparable scales.