Outlier detection basics
Outliers are values far from the rest; the IQR rule is a simple way to flag them.
In this page:
Syntax
q1 = df["column"].quantile(0.25)
q3 = df["column"].quantile(0.75)
iqr = q3 - q1
outliers = df[(df["column"] < q1 - 1.5 * iqr) | (df["column"] > q3 + 1.5 * iqr)]Outlier detection basics
Compute the first and third quartiles, then the interquartile range IQR = Q3 - Q1. Values below Q1 - 1.5*IQR or above Q3 + 1.5*IQR are commonly flagged as outliers.
Z-scores are another approach.
Investigate outliers before removing them, since they may be real.
Note:
quantile(0.25) and quantile(0.75) give the quartiles.
Example: Outlier detection basics
import pandas as pd
s = pd.Series([10, 12, 11, 13, 12, 95])
q1, q3 = s.quantile(0.25), s.quantile(0.75)
iqr = q3 - q1
low, high = q1 - 1.5 * iqr, q3 + 1.5 * iqr
print("bounds:", low, high)
print(s[(s < low) | (s > high)])
# Output:
# bounds: 9.0 15.0
# 5 95
# dtype: int64Related Topics
Common Mistakes
- Deleting outliers without checking whether they are genuine
- Applying the IQR rule to tiny samples
- Confusing outliers with errors
Chapter Summary
- Outliers are far from the bulk
- IQR rule uses 1.5 times IQR
- Investigate before dropping
- quantile gives quartiles
🔒
Chapter Quiz — Complete all 7 topics to unlock
0/7 topics done
Complete these topics first: