PySpark: Read Large CSV files efficiently

Scenario

You have a large CSV file (100GB+ of data) with millions of records. Loading the file without optimization causes memory issues and slow performance.

Solution: Use Partitioning & Parquet for Faster Processing

Step 1: Read the Large CSV in PySpark

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("OptimizeCSV").getOrCreate()

# Read CSV file with optimal options
df = spark.read.option("header", "true") \
               .option("inferSchema", "true") \
               .csv("path_location/large_data.csv")

df.display(5)

🔹 Why?

• inferSchema=True automatically detects column types.

• CSV files are slow, so we’ll convert them to Parquet.

Step 2: Convert CSV to Parquet & Partition Data

df.write.mode("overwrite") \
      .partitionBy("year") \
      .parquet("path_location/optimized_data/")

Or

df.write.mode("overwrite") \
      .format("parquet") \
      .partitionBy("year") \
      .save("path_location/optimized_data/")

• Parquet is 10x faster than CSV.

• Partitioning by "year" speeds up queries.

The large dataset is now efficiently stored & processed faster! 🚀