Reading large CSV files in chunks in Pandas
To read large CSV files in chunks in Pandas, use the
read_csv(~) method and specify the
chunksize parameter. This is particularly useful if you are facing a
MemoryError when trying to read in the whole DataFrame at once.
Consider the following
To read this file in chunks of two rows, set
chunksize like so:
for chunk in pd.read_csv("sample.txt", chunksize=2):print(chunk)print("-----")A B0 1 21 3 4-----A B2 5 63 7 8-----A B4 9 10-----
chunk is a DataFrame, allowing you to work with the dataset piece by piece if you do not need the whole dataset in memory at one time.