Important Business Use Case - Solved by ADF (Azure Data Factory)
Arpit TyagiTable of contents
- Step 1: Do we have the table customer_copy?
- Step 2: Use “Lookup Activity” for the Total Count of records:
- Step 3: Use “IF Activity” to check whether the count is more than 100 or not!!
- Step 4: Use “Copy Activity” in the True Gate:
- Step 5: Test-Run the pipeline":
- Step 6: Verify the New Table and its data:
“Use Case”: Whenever we have more than 100 records in the customer table, we copy the customer data to another table customer_copy within the SQL DB. However, whenever we do this copy, we first truncate the table 'customer_copy' and then copy the data from 'customer' table
Step 1: Do we have the table customer_copy?
Step 2: Use “Lookup Activity” for the Total Count of records:
Step 3: Use “IF Activity” to check whether the count is more than 100 or not!!
Step 4: Use “Copy Activity” in the True Gate:
Step 5: Test-Run the pipeline":
Step 6: Verify the New Table and its data:
Subscribe to my newsletter
Read articles from Arpit Tyagi directly inside your inbox. Subscribe to the newsletter, and don't miss out.
Written by
Arpit Tyagi
Arpit Tyagi
Experienced Data Engineer passionate about building and optimizing data infrastructure to fuel powerful insights and decision-making. With a deep understanding of data pipelines, ETL processes, and cloud platforms, I specialize in transforming raw data into clean, structured datasets that empower analytics and machine learning applications. My expertise includes designing scalable architectures, managing large datasets, and ensuring data quality across the entire lifecycle. I thrive on solving complex data challenges using modern tools and technologies like Azure, Tableau, Alteryx, Spark. Through this blog, I aim to share best practices, tutorials, and industry insights to help fellow data engineers and enthusiasts master the art of building data-driven solutions.