Chunking large files
Reading a big file in chunks keeps memory low by processing a slice at a time.
In this page:
Syntax
for chunk in pd.read_csv("file.csv", chunksize=n):
# process chunk (a DataFrame)Chunking large files
read_csv(chunksize=n) returns an iterator of DataFrames with n rows each. Process each chunk, such as aggregating, and combine the small results. This lets you handle files that do not fit in memory.
Note:
Also consider usecols and dtype to shrink memory before chunking.
Example: Chunking large files
import io
import pandas as pd
csv = "x\n" + "\n".join(str(i) for i in range(1, 11))
total = 0
for chunk in pd.read_csv(io.StringIO(csv), chunksize=4):
total += chunk["x"].sum()
print("chunk rows:", len(chunk))
print("total:", total)
# Output:
# chunk rows: 4
# chunk rows: 4
# chunk rows: 2
# total: 55Related Topics
Common Mistakes
- Concatenating all chunks and losing the benefit
- Forgetting the iterator is used once
- Aggregating incorrectly across chunks
Chapter Summary
- chunksize returns an iterator
- Process each chunk separately
- Combine small results
- usecols and dtype reduce memory
🔒
Chapter Quiz — Complete all 7 topics to unlock
0/7 topics done
Complete these topics first: