{"id":1029,"date":"2021-01-25T02:28:39","date_gmt":"2021-01-25T02:28:39","guid":{"rendered":"http:\/\/jackofalltradesmasterofsome.com\/blog\/?p=1029"},"modified":"2023-02-28T14:47:58","modified_gmt":"2023-02-28T14:47:58","slug":"streaming-etl-with-azure-data-factory-and-cdc-provisioning-provisioning-azure-blob-storage","status":"publish","type":"post","link":"http:\/\/jackofalltradesmasterofsome.com\/blog\/2021\/01\/25\/streaming-etl-with-azure-data-factory-and-cdc-provisioning-provisioning-azure-blob-storage\/","title":{"rendered":"Streaming ETL with Azure Data Factory and CDC – Provisioning Azure Blob Storage"},"content":{"rendered":"\n

In this series we look at building a Streaming ETL with Azure Data Factory and CDC – Provisioning Azure Blob Storage . This is Part 4, The rest of the series is below.<\/p>\n\n\n\n

  1. Enabling CDC<\/a><\/li>
  2. Setting up Audit Tables<\/a><\/li>
  3. Provisioning Azure Data Factory<\/a><\/li>
  4. Provisioning Azure Blog Storage<\/a><\/li>
  5. Create Data Source Connection in ADF<\/a><\/li>
  6. Create Incremental Pipeline in ADF<\/a><\/li>
  7. Create a Parameter Driven Pipeline<\/a><\/li>
  8. Create a Rolling Trigger<\/a><\/li><\/ol>\n\n\n\n

    Side Note: Want to learn SQL <\/a>or Python <\/a>for free. In less then 10 minutes a day and less than an hour total? Signup for my free classes delivered daily right to your email inbox for free! <\/em><\/p>\n\n\n\n

    \"\"<\/a><\/figure>\n\n\n\n
    \"\"<\/a><\/figure>\n\n\n\n

    Now back to the article…<\/p>\n\n\n\n

    This series uses the Adventureworks database. For more information on how to get that set up see my Youtube video for Downloading <\/a>and Restoring<\/a> the database. <\/p>\n\n\n\n

    Provisioning Azure Blob Storage<\/p>\n\n\n\n

    1. For this section, we are assuming you have\nalready signed up for a Microsoft Azure Account. If you have not, navigate to a\nhttps:\/\/azure.microsoft.com\/<\/a> to get\nyour free account. <\/li><\/ol>\n\n\n\n
      • Open the Azure Portal and provision a new Data Factory  <\/li><\/ul>\n\n\n\n
        \"\"<\/figure>\n\n\n\n
        • Give the storage a resource group, a name and leave the rest as is. Select Review and Create to complete set up.<\/li><\/ul>\n\n\n\n
          \"\"<\/figure>\n\n\n\n
          • One the storage is set up, go to the resource and select \u201cStorage Explorer\u201d and create a new blob container called datalake<\/li><\/ul>\n\n\n\n
            \"\"<\/figure>\n\n\n\n
            \"\"<\/figure>\n\n\n\n

            Streaming ETL with Azure Data Factory and CDC – Provisioning Azure Blob Storage<\/p>\n","protected":false},"excerpt":{"rendered":"

            In this series we look at building a Streaming ETL with Azure Data Factory and CDC – Provisioning Azure Blob Storage . This is Part 4, The rest of the series is below. Enabling CDC Setting up Audit Tables Provisioning Azure Data Factory Provisioning Azure Blog Storage Create Data Source Connection in ADF Create Incremental […]<\/p>\n","protected":false},"author":1,"featured_media":822,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[4,28,27,11,1],"tags":[104],"class_list":["post-1029","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-analytics","category-azure","category-big-data","category-data-warehouse","category-uncategorized","tag-azure-data-factory"],"_links":{"self":[{"href":"http:\/\/jackofalltradesmasterofsome.com\/blog\/wp-json\/wp\/v2\/posts\/1029","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/jackofalltradesmasterofsome.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/jackofalltradesmasterofsome.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/jackofalltradesmasterofsome.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/jackofalltradesmasterofsome.com\/blog\/wp-json\/wp\/v2\/comments?post=1029"}],"version-history":[{"count":3,"href":"http:\/\/jackofalltradesmasterofsome.com\/blog\/wp-json\/wp\/v2\/posts\/1029\/revisions"}],"predecessor-version":[{"id":1308,"href":"http:\/\/jackofalltradesmasterofsome.com\/blog\/wp-json\/wp\/v2\/posts\/1029\/revisions\/1308"}],"wp:featuredmedia":[{"embeddable":true,"href":"http:\/\/jackofalltradesmasterofsome.com\/blog\/wp-json\/wp\/v2\/media\/822"}],"wp:attachment":[{"href":"http:\/\/jackofalltradesmasterofsome.com\/blog\/wp-json\/wp\/v2\/media?parent=1029"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/jackofalltradesmasterofsome.com\/blog\/wp-json\/wp\/v2\/categories?post=1029"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/jackofalltradesmasterofsome.com\/blog\/wp-json\/wp\/v2\/tags?post=1029"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}