Streaming ETL with Azure Data Factory and CDC – Creating the Rolling ETL Window

In this series we look at building a Streaming ETL with Azure Data Factory and CDC – Creating the Rolling ETL Window. This is Part 1, The rest of the series is below.

1. Enabling CDC
2. Setting up Audit Tables
3. Provisioning Azure Data Factory
4. Provisioning Azure Blog Storage
5. Create Data Source Connection in ADF
6. Create Incremental Pipeline in ADF
7. Create a Parameter Driven Pipeline
8. Create a Rolling Trigger

This series uses the Adventureworks database. For more information on how to get that set up see my Youtube video for Downloading and Restoring the database.

Creating the Rolling ETL Window

Now that we have our parameter driven piepline we can create a trigger using a rolling time to run intermittantly and pick up changes.

1. Click on Add Trigger -> New

• Create a new trigger set as Tumbling window and set it for a time in the future to start.

• On the following screen, set the start and end dates to

@formatDateTime(trigger().outputs.windowStartTime,’yyyy-MM-dd HH:mm:ss.fff’)

@formatDateTime(trigger().outputs.windowEndTime,’yyyy-MM-dd HH:mm:ss.fff’)

Streaming ETL with Azure Data Factory and CDC – Creating the Rolling ETL Window