Duplicate rows
Duplicated rows can inflate your numbers, so learn to spot and drop them.
In this page:
Syntax
df.duplicated()
df = df.drop_duplicates()Duplicate rows
duplicated() flags repeated rows (the first occurrence is False by default). drop_duplicates() removes them, and subset limits the check to certain columns. keep="last" or keep=False changes which copies are retained.
Note:
df.duplicated().sum() counts the duplicates.
Example: Duplicate rows
import pandas as pd
df = pd.DataFrame({"id": [1, 2, 2, 3], "name": ["Ann", "Bob", "Bob", "Cy"]})
print(df.duplicated().tolist())
print(df.drop_duplicates())
print(df.drop_duplicates(subset="name", keep="last"))
# Output:
# [False, False, True, False]
# id name
# 0 1 Ann
# 1 2 Bob
# 3 3 Cy
# id name
# 0 1 Ann
# 2 2 Bob
# 3 3 CyRelated Topics
Common Mistakes
- Dropping duplicates without deciding which columns define a duplicate
- Forgetting the first occurrence is kept
- Not resetting the index after dropping
Chapter Summary
- duplicated flags repeats
- drop_duplicates removes them
- subset picks key columns
- keep chooses which copy stays
🔒
Chapter Quiz — Complete all 7 topics to unlock
0/7 topics done
Complete these topics first: