Outliers are not always errors

Outliers could be:

• True rare events (high-value orders)
• Measurement/unit problems
• Recording errors
• Different segment (VIP customers)

So the first step is always: investigate.

Common strategies

1) Investigate and correct (best)

• Check the source system
• Confirm units
• Validate against business rules

2) Remove outliers (use carefully)

clean = df[(df["amount"] >= lower) & (df["amount"] <= upper)].copy()
print(clean)

clean = df[(df["amount"] >= lower) & (df["amount"] <= upper)].copy()
print(clean)

Removing is risky when you report business totals.

3) Cap / winsorize

Capping keeps all rows but limits extreme values.

df["amount_capped"] = df["amount"].clip(lower, upper)

df["amount_capped"] = df["amount"].clip(lower, upper)

4) Transform (e.g., log)

Useful when values span many orders of magnitude.

import numpy as np
 
df["amount_log"] = np.log1p(df["amount"])  # log(1+x) to handle 0

import numpy as np
 
df["amount_log"] = np.log1p(df["amount"])  # log(1+x) to handle 0

Which strategy should you pick?

• Reporting metrics → investigate, maybe cap
• ML features → cap or transform often helps
• Fraud/anomaly detection → keep outliers (they may be the signal)

Always document

• detection rule
• chosen handling method
• expected impact

Visualize it

  flowchart LR
  A["Outlier detected"] --> B{"Why is it there?"}
  B -->|"data error"| C["Investigate & correct"]
  B -->|"true rare event"| D["Keep or flag"]
  B -->|"reporting / ML feature"| E["Cap (winsorize)"]
  B -->|"skewed magnitude"| F["Transform (log)"]

Most values cluster tightly together; an outlier sits far away from that cluster. Capping doesn’t delete it — it pulls the value back to a boundary you choose.

🧪 Try It Yourself

Exercise 1 – Cap values with clip

Exercise 2 – Cap using the sign trick

Exercise 3 – Transform with log1p

Next

Now that extreme values are under control, move on to Feature Scaling (MinMax vs Standard) so all your numeric features live on comparable scales.