{"id":1021,"date":"2021-01-27T01:20:05","date_gmt":"2021-01-27T01:20:05","guid":{"rendered":"http:\/\/jackofalltradesmasterofsome.com\/blog\/?p=1021"},"modified":"2021-02-05T14:37:35","modified_gmt":"2021-02-05T14:37:35","slug":"streaming-etl-with-azure-data-factory-and-cdc-creating-a-data-source-connection-in-azure-data-factory","status":"publish","type":"post","link":"https:\/\/jackofalltradesmasterofsome.com\/blog\/2021\/01\/27\/streaming-etl-with-azure-data-factory-and-cdc-creating-a-data-source-connection-in-azure-data-factory\/","title":{"rendered":"Streaming ETL with Azure Data Factory and CDC – Creating a Data Source Connection in Azure Data Factory"},"content":{"rendered":"\n

In this series we look at building a Streaming ETL with Azure Data Factory and CDC – Creating a Data Source Connection in Azure Data Factory. This is Part 5, The rest of the series is below.<\/p>\n\n\n\n

  1. Enabling CDC<\/a><\/li>
  2. Setting up Audit Tables<\/a><\/li>
  3. Provisioning Azure Data Factory<\/a><\/li>
  4. Provisioning Azure Blog Storage<\/a><\/li>
  5. Create Data Source Connection in ADF<\/a><\/li>
  6. Create Incremental Pipeline in ADF<\/a><\/li>
  7. Create a Parameter Driven Pipeline<\/a><\/li>
  8. Create a Rolling Trigger<\/a><\/li><\/ol>\n\n\n\n

    This series uses the Adventureworks database. For more information on how to get that set up see my Youtube video for Downloading <\/a>and Restoring<\/a> the database. <\/p>\n\n\n\n

    1. Create a new Dataset as a SQL Server.<\/li><\/ol>\n\n\n\n
      \"\"<\/figure>\n\n\n\n
      \"\"<\/figure>\n\n\n\n
      • Name it DimProperty and select your Integrated Runtime for the local SQL server. For the table, select the CDC table.<\/li><\/ul>\n\n\n\n
        \"\"<\/figure>\n\n\n\n
        • Create a new Dataset and this time select Azure Blob Storage and a DelimitedText.<\/li><\/ul>\n\n\n\n
          \"\"<\/figure>\n\n\n\n
          \"\"<\/figure>\n\n\n\n
          • Name the csv_DimProperty and select New Linked Service.<\/li><\/ul>\n\n\n\n
            \"\"<\/figure>\n\n\n\n
            • Name the blob storage \u201cDataLake\u201d to match your storage account and point it to your storage account in the subscription.<\/li><\/ul>\n\n\n\n
              \"\"<\/figure>\n\n\n\n
              • Select the \u201cdatalake\u201d from the file folder section or type it in and set first row as header and select ok to complete.<\/li><\/ul>\n\n\n\n
                \"\"<\/figure>\n\n\n\n
                • We should now have our two datasets in the resource\u2019s sections for the property CDC transfer. Select Publish All at the top to save your changes<\/li><\/ul>\n\n\n\n
                  \"\"<\/figure>\n\n\n\n

                  Streaming ETL with Azure Data Factory and CDC – Creating a Data Source Connection in Azure Data Factory<\/p>\n","protected":false},"excerpt":{"rendered":"

                  In this series we look at building a Streaming ETL with Azure Data Factory and CDC – Creating a Data Source Connection in Azure Data Factory. This is Part 5, The rest of the series is below. Enabling CDC Setting up Audit Tables Provisioning Azure Data Factory Provisioning Azure Blog Storage Create Data Source Connection […]<\/p>\n","protected":false},"author":1,"featured_media":822,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[4,28,27,11,1],"tags":[104],"class_list":["post-1021","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-analytics","category-azure","category-big-data","category-data-warehouse","category-uncategorized","tag-azure-data-factory"],"_links":{"self":[{"href":"https:\/\/jackofalltradesmasterofsome.com\/blog\/wp-json\/wp\/v2\/posts\/1021","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/jackofalltradesmasterofsome.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/jackofalltradesmasterofsome.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/jackofalltradesmasterofsome.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/jackofalltradesmasterofsome.com\/blog\/wp-json\/wp\/v2\/comments?post=1021"}],"version-history":[{"count":2,"href":"https:\/\/jackofalltradesmasterofsome.com\/blog\/wp-json\/wp\/v2\/posts\/1021\/revisions"}],"predecessor-version":[{"id":1081,"href":"https:\/\/jackofalltradesmasterofsome.com\/blog\/wp-json\/wp\/v2\/posts\/1021\/revisions\/1081"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/jackofalltradesmasterofsome.com\/blog\/wp-json\/wp\/v2\/media\/822"}],"wp:attachment":[{"href":"https:\/\/jackofalltradesmasterofsome.com\/blog\/wp-json\/wp\/v2\/media?parent=1021"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/jackofalltradesmasterofsome.com\/blog\/wp-json\/wp\/v2\/categories?post=1021"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/jackofalltradesmasterofsome.com\/blog\/wp-json\/wp\/v2\/tags?post=1021"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}