Build your first ETL pipeline using PySpark

Overview

In this session, we’ll walk through building a simple ETL (Extract, Transform, Load) pipeline using PySpark

Steps Involved

1. Setup Spark Environment

2. Read Data from CSV

3. Cleanse and Transform the CSV data

4. Write final results to target location

Setup Spark Environment

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("SampleETL").getOrCreate()

Import Libraries: The pyspark.sql library is imported to use the SparkSession class, which is essential for any Spark application

Read Data from CSV

•   input_path="stroage_path_to_the_location"
    df = spark.read.option("header","true").csv(input_path)
    display(df)
  

Cleanse and Transform Data

• Drop Null Values

    df_cleansed=df.dropna(["Column 1","Column 2"])
  

• Filter Columns

    from pyspark.sql.functions import lit
    df_filtered=df_cleansed.where("Column 3 == lit('TEXT') ")
  

• Add New Columns

    df_addition=df_filtered.withColumn("CreateDt", current_timestamp())
  

• Write Data to Target Location

    taget_path_location=f"/mnt/silver/storage1/container_location"
    df_addition.write.mode("overwrite").format("parquet").save(f"{taget_path_location}")