We can achieve the incremental loading using the spark data partition.
I will be demonstrating with a simple example here, and you can you according to the business need.
Let's consider the following data frame.(This could be your csv, or any formatt...
