{"id":1024,"date":"2021-01-27T01:33:06","date_gmt":"2021-01-27T01:33:06","guid":{"rendered":"http:\/\/jackofalltradesmasterofsome.com\/blog\/?p=1024"},"modified":"2021-02-05T14:34:49","modified_gmt":"2021-02-05T14:34:49","slug":"streaming-etl-with-azure-data-factory-and-cdc-creating-the-rolling-etl-window","status":"publish","type":"post","link":"https:\/\/jackofalltradesmasterofsome.com\/blog\/2021\/01\/27\/streaming-etl-with-azure-data-factory-and-cdc-creating-the-rolling-etl-window\/","title":{"rendered":"Streaming ETL with Azure Data Factory and CDC – Creating the Rolling ETL Window"},"content":{"rendered":"\n

In this series we look at building a Streaming ETL with Azure Data Factory and CDC – Creating the Rolling ETL Window. This is Part 1, The rest of the series is below.<\/p>\n\n\n\n

  1. Enabling CDC<\/a><\/li>
  2. Setting up Audit Tables<\/a><\/li>
  3. Provisioning Azure Data Factory<\/a><\/li>
  4. Provisioning Azure Blog Storage<\/a><\/li>
  5. Create Data Source Connection in ADF<\/a><\/li>
  6. Create Incremental Pipeline in ADF<\/a><\/li>
  7. Create a Parameter Driven Pipeline<\/a><\/li>
  8. Create a Rolling Trigger<\/a><\/li><\/ol>\n\n\n\n

    This series uses the Adventureworks database. For more information on how to get that set up see my Youtube video for Downloading <\/a>and Restoring<\/a> the database. <\/p>\n\n\n\n

    Creating the Rolling ETL Window<\/p>\n\n\n\n

    Now that we have our\nparameter driven piepline we can create a trigger using a rolling time to run\nintermittantly and pick up changes. <\/p>\n\n\n\n

    1. Click on Add Trigger -> New<\/li><\/ol>\n\n\n\n
      \"\"<\/figure>\n\n\n\n
      • Create a new trigger set as Tumbling window and set it for a time in the future to start.<\/li><\/ul>\n\n\n\n
        \"\"<\/figure>\n\n\n\n
        • On the following screen, set the start and end\ndates to <\/li><\/ul>\n\n\n\n

          @formatDateTime(trigger().outputs.windowStartTime,’yyyy-MM-dd\nHH:mm:ss.fff’)<\/p>\n\n\n\n

          @formatDateTime(trigger().outputs.windowEndTime,’yyyy-MM-dd HH:mm:ss.fff’)<\/p>\n\n\n\n

          \"\"<\/figure>\n\n\n\n

          Streaming ETL with Azure Data Factory and CDC – Creating the Rolling ETL Window<\/p>\n","protected":false},"excerpt":{"rendered":"

          In this series we look at building a Streaming ETL with Azure Data Factory and CDC – Creating the Rolling ETL Window. This is Part 1, The rest of the series is below. Enabling CDC Setting up Audit Tables Provisioning Azure Data Factory Provisioning Azure Blog Storage Create Data Source Connection in ADF Create Incremental […]<\/p>\n","protected":false},"author":1,"featured_media":822,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[4,28,27,11],"tags":[104],"class_list":["post-1024","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-analytics","category-azure","category-big-data","category-data-warehouse","tag-azure-data-factory"],"_links":{"self":[{"href":"https:\/\/jackofalltradesmasterofsome.com\/blog\/wp-json\/wp\/v2\/posts\/1024","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/jackofalltradesmasterofsome.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/jackofalltradesmasterofsome.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/jackofalltradesmasterofsome.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/jackofalltradesmasterofsome.com\/blog\/wp-json\/wp\/v2\/comments?post=1024"}],"version-history":[{"count":2,"href":"https:\/\/jackofalltradesmasterofsome.com\/blog\/wp-json\/wp\/v2\/posts\/1024\/revisions"}],"predecessor-version":[{"id":1078,"href":"https:\/\/jackofalltradesmasterofsome.com\/blog\/wp-json\/wp\/v2\/posts\/1024\/revisions\/1078"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/jackofalltradesmasterofsome.com\/blog\/wp-json\/wp\/v2\/media\/822"}],"wp:attachment":[{"href":"https:\/\/jackofalltradesmasterofsome.com\/blog\/wp-json\/wp\/v2\/media?parent=1024"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/jackofalltradesmasterofsome.com\/blog\/wp-json\/wp\/v2\/categories?post=1024"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/jackofalltradesmasterofsome.com\/blog\/wp-json\/wp\/v2\/tags?post=1024"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}