{"id":634,"date":"2020-04-13T19:20:49","date_gmt":"2020-04-13T19:20:49","guid":{"rendered":"http:\/\/jackofalltradesmasterofsome.com\/blog\/?p=634"},"modified":"2020-04-14T17:26:29","modified_gmt":"2020-04-14T17:26:29","slug":"modern-data-architecture-part-5-loading-data-lake-with-data-factory","status":"publish","type":"post","link":"https:\/\/jackofalltradesmasterofsome.com\/blog\/2020\/04\/13\/modern-data-architecture-part-5-loading-data-lake-with-data-factory\/","title":{"rendered":"Modern Data Architecture \u2013 Part 5 &#8211; Loading Data Lake with Data Factory"},"content":{"rendered":"\n<p>Modern Data Architecture \u2013 Part 5 &#8211; Loading Data Lake with Data Factory<\/p>\n\n\n\n<p>The Azure Data Factory (ADF) is a service designed to allow developers to integrate disparate data sources.&nbsp; It is a platform somewhat like SSIS in the cloud to manage the data you have both on-prem and in the cloud.<\/p>\n\n\n\n<p>It provides\naccess to on-premises data in SQL Server and cloud data in Azure Storage (Blob\nand Tables) and Azure SQL Database.&nbsp;\nAccess to on-premises data is provided through a data management gateway\nthat connects to on-premises SQL Server databases.<\/p>\n\n\n\n<p>It is not a\ndrag-and-drop interface like SSIS.&nbsp;\nInstead, data processing is enabled initially through Hive, Pig and\ncustom C# activities.&nbsp; Such activities\ncan be used to clean data, mask data fields, and transform data in a wide\nvariety of complex ways.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Creating a Data Factory Resource<\/h3>\n\n\n\n<ol class=\"wp-block-list\"><li>In the search bar, search for \u201cData Factories\u201d<\/li><\/ol>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"445\" height=\"206\" src=\"http:\/\/jackofalltradesmasterofsome.com\/blog\/wp-content\/uploads\/2020\/04\/image-32.png\" alt=\"\" class=\"wp-image-679\" srcset=\"https:\/\/jackofalltradesmasterofsome.com\/blog\/wp-content\/uploads\/2020\/04\/image-32.png 445w, https:\/\/jackofalltradesmasterofsome.com\/blog\/wp-content\/uploads\/2020\/04\/image-32-300x139.png 300w\" sizes=\"auto, (max-width: 445px) 100vw, 445px\" \/><\/figure>\n\n\n\n<ul class=\"wp-block-list\"><li>Select \u201c+ Add\u201d to create a new Data Factory<\/li><li>Fill in the necessary details for your data factory and select \u201cCreate\u201d<ul><li>Name \u2013 \u201ctraining-datafactories-<em>yourname<\/em>\u201d<ul><li>Name does not allow underscores, use dashes instead<\/li><\/ul><\/li><\/ul><ul><li>Version &#8211; V2<\/li><\/ul><ul><li>Subscription \u2013 The default subscription you set up for the demo<\/li><\/ul><ul><li>Resource Group \u201ctraining_resourcegroup_<em>yourname<\/em>\u201d<\/li><\/ul><ul><li>Location \u2013 \u201cEast US 2\u201d or the same as your resource group<\/li><\/ul><ul><li>Enable Git \u2013 Uncheck this for the demo. In real development, you will want to link your code to a repository for disaster recovery.<\/li><\/ul><\/li><\/ul>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"390\" height=\"341\" src=\"http:\/\/jackofalltradesmasterofsome.com\/blog\/wp-content\/uploads\/2020\/04\/image-33.png\" alt=\"\" class=\"wp-image-680\" srcset=\"https:\/\/jackofalltradesmasterofsome.com\/blog\/wp-content\/uploads\/2020\/04\/image-33.png 390w, https:\/\/jackofalltradesmasterofsome.com\/blog\/wp-content\/uploads\/2020\/04\/image-33-300x262.png 300w\" sizes=\"auto, (max-width: 390px) 100vw, 390px\" \/><\/figure>\n\n\n\n<ul class=\"wp-block-list\"><li>After a short loading period, your Data Factory resource will be provisioned and listed in your dashboard.<\/li><\/ul>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"530\" height=\"205\" src=\"http:\/\/jackofalltradesmasterofsome.com\/blog\/wp-content\/uploads\/2020\/04\/image-34.png\" alt=\"\" class=\"wp-image-681\" srcset=\"https:\/\/jackofalltradesmasterofsome.com\/blog\/wp-content\/uploads\/2020\/04\/image-34.png 530w, https:\/\/jackofalltradesmasterofsome.com\/blog\/wp-content\/uploads\/2020\/04\/image-34-300x116.png 300w\" sizes=\"auto, (max-width: 530px) 100vw, 530px\" \/><\/figure>\n\n\n\n<ul class=\"wp-block-list\"><li>Click on the new Data Factory Resource to view its details. Select the \u201cAuthor and Monitor\u201d button to navigate to the editor.<\/li><\/ul>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"536\" height=\"280\" src=\"http:\/\/jackofalltradesmasterofsome.com\/blog\/wp-content\/uploads\/2020\/04\/image-35.png\" alt=\"\" class=\"wp-image-682\" srcset=\"https:\/\/jackofalltradesmasterofsome.com\/blog\/wp-content\/uploads\/2020\/04\/image-35.png 536w, https:\/\/jackofalltradesmasterofsome.com\/blog\/wp-content\/uploads\/2020\/04\/image-35-300x157.png 300w\" sizes=\"auto, (max-width: 536px) 100vw, 536px\" \/><\/figure>\n\n\n\n<h3 class=\"wp-block-heading\">Setting Up a Data Source<\/h3>\n\n\n\n<ol class=\"wp-block-list\"><li>Select \u201cCreate a Pipeline\u201d from the editor. In pipelines we can create data flows between data sources and data destinations. For this example, we will be loading data locally stored into our newly provisioned data lake and scheduling the job to run on a schedule. In real world scenarios, the data source could be a FTP file path or a shared drive. We will also create a data source from a SQL database to the data lake as we would in a real world scenario.<\/li><li>Select Datasets-&gt;New dataset.<\/li><\/ol>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"329\" height=\"117\" src=\"http:\/\/jackofalltradesmasterofsome.com\/blog\/wp-content\/uploads\/2020\/04\/image-36.png\" alt=\"\" class=\"wp-image-683\" srcset=\"https:\/\/jackofalltradesmasterofsome.com\/blog\/wp-content\/uploads\/2020\/04\/image-36.png 329w, https:\/\/jackofalltradesmasterofsome.com\/blog\/wp-content\/uploads\/2020\/04\/image-36-300x107.png 300w\" sizes=\"auto, (max-width: 329px) 100vw, 329px\" \/><\/figure>\n\n\n\n<ul class=\"wp-block-list\"><li>In the new datasets section, select \u201cSQL Server\u201d <\/li><\/ul>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"282\" height=\"261\" src=\"http:\/\/jackofalltradesmasterofsome.com\/blog\/wp-content\/uploads\/2020\/04\/image-37.png\" alt=\"\" class=\"wp-image-684\"\/><\/figure>\n\n\n\n<ul class=\"wp-block-list\"><li>In Set Properties set the name as \u201cSource_Database\u201d and select a + New Linked Service<\/li><\/ul>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"457\" height=\"192\" src=\"http:\/\/jackofalltradesmasterofsome.com\/blog\/wp-content\/uploads\/2020\/04\/image-38.png\" alt=\"\" class=\"wp-image-685\" srcset=\"https:\/\/jackofalltradesmasterofsome.com\/blog\/wp-content\/uploads\/2020\/04\/image-38.png 457w, https:\/\/jackofalltradesmasterofsome.com\/blog\/wp-content\/uploads\/2020\/04\/image-38-300x126.png 300w\" sizes=\"auto, (max-width: 457px) 100vw, 457px\" \/><\/figure>\n\n\n\n<ul class=\"wp-block-list\"><li>Complete the remaining details for the SQL Data Connection<ul><li>Name \u201cTraining_SQLDatabase_<em>yourname<\/em>\u201d<\/li><\/ul><ul><li>Connect via integration run time \u2013 Leave as default \u201cAutoResolveIntegationRuntime\u201d<\/li><\/ul><ul><li>Connection String<ul><li>From Azure Subscription <\/li><\/ul><ul><li>Server name &#8211; \u201ctraining-sqlserver-<em>yourname<\/em>\u201d <ul><li>Enter all the remaining database credentials from the previous lab and select \u201cTest Connection\u201d to test.<\/li><\/ul><ul><li>If you receive a connection error based on IP Firewall, record the IP address and whitelist it on the database similar to how it was done in lab 4.<\/li><\/ul><\/li><\/ul><\/li><\/ul><\/li><\/ul>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"348\" height=\"333\" src=\"http:\/\/jackofalltradesmasterofsome.com\/blog\/wp-content\/uploads\/2020\/04\/image-39.png\" alt=\"\" class=\"wp-image-686\" srcset=\"https:\/\/jackofalltradesmasterofsome.com\/blog\/wp-content\/uploads\/2020\/04\/image-39.png 348w, https:\/\/jackofalltradesmasterofsome.com\/blog\/wp-content\/uploads\/2020\/04\/image-39-300x287.png 300w\" sizes=\"auto, (max-width: 348px) 100vw, 348px\" \/><\/figure>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"346\" height=\"353\" src=\"http:\/\/jackofalltradesmasterofsome.com\/blog\/wp-content\/uploads\/2020\/04\/image-40.png\" alt=\"\" class=\"wp-image-687\" srcset=\"https:\/\/jackofalltradesmasterofsome.com\/blog\/wp-content\/uploads\/2020\/04\/image-40.png 346w, https:\/\/jackofalltradesmasterofsome.com\/blog\/wp-content\/uploads\/2020\/04\/image-40-294x300.png 294w\" sizes=\"auto, (max-width: 346px) 100vw, 346px\" \/><\/figure>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"222\" height=\"364\" src=\"http:\/\/jackofalltradesmasterofsome.com\/blog\/wp-content\/uploads\/2020\/04\/image-41.png\" alt=\"\" class=\"wp-image-688\" srcset=\"https:\/\/jackofalltradesmasterofsome.com\/blog\/wp-content\/uploads\/2020\/04\/image-41.png 222w, https:\/\/jackofalltradesmasterofsome.com\/blog\/wp-content\/uploads\/2020\/04\/image-41-183x300.png 183w\" sizes=\"auto, (max-width: 222px) 100vw, 222px\" \/><\/figure>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"513\" height=\"116\" src=\"http:\/\/jackofalltradesmasterofsome.com\/blog\/wp-content\/uploads\/2020\/04\/image-42.png\" alt=\"\" class=\"wp-image-689\" srcset=\"https:\/\/jackofalltradesmasterofsome.com\/blog\/wp-content\/uploads\/2020\/04\/image-42.png 513w, https:\/\/jackofalltradesmasterofsome.com\/blog\/wp-content\/uploads\/2020\/04\/image-42-300x68.png 300w\" sizes=\"auto, (max-width: 513px) 100vw, 513px\" \/><\/figure>\n\n\n\n<ul class=\"wp-block-list\"><li>In the last properties window, set the Table Name to \u201csales.customers\u201d.<\/li><\/ul>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"450\" height=\"232\" src=\"http:\/\/jackofalltradesmasterofsome.com\/blog\/wp-content\/uploads\/2020\/04\/image-43.png\" alt=\"\" class=\"wp-image-690\" srcset=\"https:\/\/jackofalltradesmasterofsome.com\/blog\/wp-content\/uploads\/2020\/04\/image-43.png 450w, https:\/\/jackofalltradesmasterofsome.com\/blog\/wp-content\/uploads\/2020\/04\/image-43-300x155.png 300w\" sizes=\"auto, (max-width: 450px) 100vw, 450px\" \/><\/figure>\n\n\n\n<ul class=\"wp-block-list\"><li>Click\n\u201cOK\u201d to save.<\/li><\/ul>\n\n\n\n<ul class=\"wp-block-list\"><li>Create a new connection, but this time select \u201cAzure Blog Storage\u201d<\/li><\/ul>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"292\" height=\"271\" src=\"http:\/\/jackofalltradesmasterofsome.com\/blog\/wp-content\/uploads\/2020\/04\/image-44.png\" alt=\"\" class=\"wp-image-691\"\/><\/figure>\n\n\n\n<ul class=\"wp-block-list\"><li>Select Comma Delimeted and name set it as \u201cDestination_CSV\u201d and select a + New Linked Service<\/li><\/ul>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"355\" height=\"163\" src=\"http:\/\/jackofalltradesmasterofsome.com\/blog\/wp-content\/uploads\/2020\/04\/image-45.png\" alt=\"\" class=\"wp-image-692\" srcset=\"https:\/\/jackofalltradesmasterofsome.com\/blog\/wp-content\/uploads\/2020\/04\/image-45.png 355w, https:\/\/jackofalltradesmasterofsome.com\/blog\/wp-content\/uploads\/2020\/04\/image-45-300x138.png 300w\" sizes=\"auto, (max-width: 355px) 100vw, 355px\" \/><\/figure>\n\n\n\n<ol class=\"wp-block-list\"><li>In the New linked servers window enter the necessary details<ol><li>Name \u2013 \u201ctraining_storage_<em>yourname<\/em>\u201d<\/li><\/ol><ol><li>Connect via integration run time \u2013 Leave as default \u201cAutoResolveIntegationRuntime\u201d<\/li><\/ol><ol><li>Authentication method \u2013 Account Key<\/li><\/ol><ol><li>Account selection method \u2013 From Azure Subscription<ol><li>Select the training storage you created in earlier labs<\/li><\/ol><\/li><\/ol><\/li><\/ol>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"354\" height=\"359\" src=\"http:\/\/jackofalltradesmasterofsome.com\/blog\/wp-content\/uploads\/2020\/04\/image-46.png\" alt=\"\" class=\"wp-image-693\" srcset=\"https:\/\/jackofalltradesmasterofsome.com\/blog\/wp-content\/uploads\/2020\/04\/image-46.png 354w, https:\/\/jackofalltradesmasterofsome.com\/blog\/wp-content\/uploads\/2020\/04\/image-46-296x300.png 296w\" sizes=\"auto, (max-width: 354px) 100vw, 354px\" \/><\/figure>\n\n\n\n<ul class=\"wp-block-list\"><li>Once\ncomplete, select test connection and create. <\/li><\/ul>\n\n\n\n<ul class=\"wp-block-list\"><li>On the set properties, set a file path called \u201croot\/raw\u201d<\/li><\/ul>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"339\" height=\"207\" src=\"http:\/\/jackofalltradesmasterofsome.com\/blog\/wp-content\/uploads\/2020\/04\/image-47.png\" alt=\"\" class=\"wp-image-694\" srcset=\"https:\/\/jackofalltradesmasterofsome.com\/blog\/wp-content\/uploads\/2020\/04\/image-47.png 339w, https:\/\/jackofalltradesmasterofsome.com\/blog\/wp-content\/uploads\/2020\/04\/image-47-300x183.png 300w\" sizes=\"auto, (max-width: 339px) 100vw, 339px\" \/><\/figure>\n\n\n\n<ul class=\"wp-block-list\"><li>Set \u201cFirst Row as Header\u201d and Ok to save. <\/li><li>Navigate to the connections tab in the details and select \u201cTest Connection\u201d to test.<\/li><\/ul>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"529\" height=\"127\" src=\"http:\/\/jackofalltradesmasterofsome.com\/blog\/wp-content\/uploads\/2020\/04\/image-48.png\" alt=\"\" class=\"wp-image-695\" srcset=\"https:\/\/jackofalltradesmasterofsome.com\/blog\/wp-content\/uploads\/2020\/04\/image-48.png 529w, https:\/\/jackofalltradesmasterofsome.com\/blog\/wp-content\/uploads\/2020\/04\/image-48-300x72.png 300w\" sizes=\"auto, (max-width: 529px) 100vw, 529px\" \/><\/figure>\n\n\n\n<h3 class=\"wp-block-heading\">Create a Data Transfer <\/h3>\n\n\n\n<ol class=\"wp-block-list\"><li>From the Pipeline tab, In the \u201cMove and Transform\u201d section in the Activities, drag and drop a \u201cCopy Data\u201d task to the main window. In the properties editor below, select \u201cSource\u201d and select \u201cSource_Database\u201d<\/li><\/ol>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"515\" height=\"238\" src=\"http:\/\/jackofalltradesmasterofsome.com\/blog\/wp-content\/uploads\/2020\/04\/image-49.png\" alt=\"\" class=\"wp-image-696\" srcset=\"https:\/\/jackofalltradesmasterofsome.com\/blog\/wp-content\/uploads\/2020\/04\/image-49.png 515w, https:\/\/jackofalltradesmasterofsome.com\/blog\/wp-content\/uploads\/2020\/04\/image-49-300x139.png 300w\" sizes=\"auto, (max-width: 515px) 100vw, 515px\" \/><\/figure>\n\n\n\n<ul class=\"wp-block-list\"><li>Select \u201cQuery\u201d and write the following query. Use the Preview Data to validate the query works as expected.<\/li><\/ul>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"513\" height=\"160\" src=\"http:\/\/jackofalltradesmasterofsome.com\/blog\/wp-content\/uploads\/2020\/04\/image-50.png\" alt=\"\" class=\"wp-image-697\" srcset=\"https:\/\/jackofalltradesmasterofsome.com\/blog\/wp-content\/uploads\/2020\/04\/image-50.png 513w, https:\/\/jackofalltradesmasterofsome.com\/blog\/wp-content\/uploads\/2020\/04\/image-50-300x94.png 300w\" sizes=\"auto, (max-width: 513px) 100vw, 513px\" \/><\/figure>\n\n\n\n<ul class=\"wp-block-list\"><li>Click on \u201cSink\u201d and select \u201cDestination_CSV\u201d<\/li><\/ul>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"624\" height=\"206\" src=\"http:\/\/jackofalltradesmasterofsome.com\/blog\/wp-content\/uploads\/2020\/04\/image-51.png\" alt=\"\" class=\"wp-image-698\" srcset=\"https:\/\/jackofalltradesmasterofsome.com\/blog\/wp-content\/uploads\/2020\/04\/image-51.png 624w, https:\/\/jackofalltradesmasterofsome.com\/blog\/wp-content\/uploads\/2020\/04\/image-51-300x99.png 300w\" sizes=\"auto, (max-width: 624px) 100vw, 624px\" \/><\/figure>\n\n\n\n<h3 class=\"wp-block-heading\">Publishing and Running<\/h3>\n\n\n\n<ol class=\"wp-block-list\"><li>Select publish at the top of the window. <\/li><\/ol>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"507\" height=\"146\" src=\"http:\/\/jackofalltradesmasterofsome.com\/blog\/wp-content\/uploads\/2020\/04\/image-52.png\" alt=\"\" class=\"wp-image-699\" srcset=\"https:\/\/jackofalltradesmasterofsome.com\/blog\/wp-content\/uploads\/2020\/04\/image-52.png 507w, https:\/\/jackofalltradesmasterofsome.com\/blog\/wp-content\/uploads\/2020\/04\/image-52-300x86.png 300w\" sizes=\"auto, (max-width: 507px) 100vw, 507px\" \/><\/figure>\n\n\n\n<ul class=\"wp-block-list\"><li>Select \u201cdebug\u201d to run the job.<\/li><li>Once the job has succeeded, navigate back to your Storage account and view the storage browser. The new file \u201ccustomer.csv\u201d should now be available.<\/li><\/ul>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"624\" height=\"256\" src=\"http:\/\/jackofalltradesmasterofsome.com\/blog\/wp-content\/uploads\/2020\/04\/image-53.png\" alt=\"\" class=\"wp-image-700\" srcset=\"https:\/\/jackofalltradesmasterofsome.com\/blog\/wp-content\/uploads\/2020\/04\/image-53.png 624w, https:\/\/jackofalltradesmasterofsome.com\/blog\/wp-content\/uploads\/2020\/04\/image-53-300x123.png 300w\" sizes=\"auto, (max-width: 624px) 100vw, 624px\" \/><\/figure>\n\n\n\n<h3 class=\"wp-block-heading\">Setting up the job to run automatically<\/h3>\n\n\n\n<ol class=\"wp-block-list\"><li>Back in Azure Data Factory, select \u201cAdd Trigger-&gt;New Edit\u201d from your Pipeline<\/li><\/ol>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"456\" height=\"184\" src=\"http:\/\/jackofalltradesmasterofsome.com\/blog\/wp-content\/uploads\/2020\/04\/image-54.png\" alt=\"\" class=\"wp-image-701\" srcset=\"https:\/\/jackofalltradesmasterofsome.com\/blog\/wp-content\/uploads\/2020\/04\/image-54.png 456w, https:\/\/jackofalltradesmasterofsome.com\/blog\/wp-content\/uploads\/2020\/04\/image-54-300x121.png 300w\" sizes=\"auto, (max-width: 456px) 100vw, 456px\" \/><\/figure>\n\n\n\n<ul class=\"wp-block-list\"><li>Create a new trigger<ul><li>Type &#8211; Schedule <\/li><\/ul><ul><li>Start Date \u2013 Todays Date<\/li><\/ul><ul><li>Reoccurrence \u2013 Every 1 Month (We don\u2019t need this to run as frequently yet)<\/li><\/ul><ul><li>End No End<\/li><\/ul><\/li><li>Select \u201cOK to save and \u201cPublish\u201d to push your changes. <\/li><li>From the triggers menu at the bottom, you can now see your new trigger and activate and deactivate it manually.<\/li><\/ul>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"338\" height=\"241\" src=\"http:\/\/jackofalltradesmasterofsome.com\/blog\/wp-content\/uploads\/2020\/04\/image-55.png\" alt=\"\" class=\"wp-image-702\" srcset=\"https:\/\/jackofalltradesmasterofsome.com\/blog\/wp-content\/uploads\/2020\/04\/image-55.png 338w, https:\/\/jackofalltradesmasterofsome.com\/blog\/wp-content\/uploads\/2020\/04\/image-55-300x214.png 300w\" sizes=\"auto, (max-width: 338px) 100vw, 338px\" \/><\/figure>\n\n\n\n<p> Be sure to check out my full online class on the topic. A hands on walk through of a Modern Data Architecture using Microsoft Azure. For beginners and experienced business intelligence experts alike, learn the basic of navigating the Azure Portal to building an end to end solution of a modern data warehouse using popular technologies such as SQL\u00a0Database, Data Lake, Data Factory, Data Bricks, Azure Synapse Data Warehouse and Power BI. Link to the class can be <a href=\"http:\/\/jackofalltradesmasterofsome.com\/blog\/2020\/04\/08\/modern-data-architecture-using-microsoft-azure-online-class-and-free-ebook\/\">found here<\/a> or directly <a href=\"https:\/\/www.udemy.com\/course\/modern-data-architecture-using-microsoft-azure\/learn\/lecture\/18527998#overview\">here<\/a>.  <\/p>\n\n\n<p><br>\n<br>\n<!--StartFragment--><\/p>\n\n\n<p><a href=\"http:\/\/jackofalltradesmasterofsome.com\/blog\/2020\/04\/08\/modern-data-architecture-part-1-navigating-the-azure-portal\/\">Part 1 &#8211; Navigating the Azure Portal<\/a><\/p>\n\n\n\n<p><a href=\"http:\/\/jackofalltradesmasterofsome.com\/blog\/2020\/04\/13\/modern-data-architecture-part-2-resource-groups-and-subscriptions\/\">Part 2 &#8211; Resource Groups and Subscriptions<\/a><\/p>\n\n\n\n<p><a href=\"http:\/\/jackofalltradesmasterofsome.com\/blog\/2020\/04\/13\/modern-data-architecture-part-3-creating-data-lake-storage\/\">Part 3 &#8211; Creating Data Lake Storage<\/a><\/p>\n\n\n\n<p><a href=\"http:\/\/jackofalltradesmasterofsome.com\/blog\/2020\/04\/13\/modern-data-architecture-part-4-setting-up-a-sql-server\/\">Part 4 &#8211; Setting up an Azure SQL Server<\/a><\/p>\n\n\n\n<p><a href=\"http:\/\/jackofalltradesmasterofsome.com\/blog\/2020\/04\/13\/modern-data-architecture-part-5-loading-data-lake-with-data-factory\/\">Part 5 &#8211; Loading Data Lake with Azure Data Factory<\/a><\/p>\n\n\n\n<p><a href=\"http:\/\/jackofalltradesmasterofsome.com\/blog\/2020\/04\/13\/modern-data-architecture-part-6-configuring-and-setting-up-data-bricks\/\">Part 6 &#8211; Configuring and Setting up Data Bricks<\/a><\/p>\n\n\n\n<p><a href=\"http:\/\/jackofalltradesmasterofsome.com\/blog\/2020\/04\/13\/modern-data-architecture-part-7-staging-data-into-data-lake\/\">Part 7 &#8211; Staging data into Data Lake<\/a><\/p>\n\n\n\n<p><a href=\"http:\/\/jackofalltradesmasterofsome.com\/blog\/2020\/04\/14\/modern-data-architecture-part-8-provisioning-a-synapsis-sql-data-warehouse\/\">Part 8 = Provisioning a Synapse SQL Data Warehouse<\/a><\/p>\n\n\n\n<p><a href=\"http:\/\/jackofalltradesmasterofsome.com\/blog\/2020\/04\/14\/modern-data-architecture-part-9-loading-data-synapse-data-warehouse\/\">Part 9 &#8211; Loading Data into Azure Data Synapse Data Warehouse<\/a><\/p>\n\n\n<p><!--EndFragment--><br>\n<br>\n<\/p>\n\n\n<p> Modern Data Architecture \u2013 Part 5 &#8211; Loading Data Lake with Data Factory <\/p>\n","protected":false},"excerpt":{"rendered":"<p>Modern Data Architecture \u2013 Part 5 &#8211; Loading Data Lake with Data Factory The Azure Data Factory (ADF) is a service designed to allow developers to integrate disparate data sources.&nbsp; It is a platform somewhat like SSIS in the cloud to manage the data you have both on-prem and in the cloud. It provides access [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":624,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[4,28,11,85],"tags":[],"class_list":["post-634","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-analytics","category-azure","category-data-warehouse","category-sql-server"],"_links":{"self":[{"href":"https:\/\/jackofalltradesmasterofsome.com\/blog\/wp-json\/wp\/v2\/posts\/634","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/jackofalltradesmasterofsome.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/jackofalltradesmasterofsome.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/jackofalltradesmasterofsome.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/jackofalltradesmasterofsome.com\/blog\/wp-json\/wp\/v2\/comments?post=634"}],"version-history":[{"count":6,"href":"https:\/\/jackofalltradesmasterofsome.com\/blog\/wp-json\/wp\/v2\/posts\/634\/revisions"}],"predecessor-version":[{"id":768,"href":"https:\/\/jackofalltradesmasterofsome.com\/blog\/wp-json\/wp\/v2\/posts\/634\/revisions\/768"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/jackofalltradesmasterofsome.com\/blog\/wp-json\/wp\/v2\/media\/624"}],"wp:attachment":[{"href":"https:\/\/jackofalltradesmasterofsome.com\/blog\/wp-json\/wp\/v2\/media?parent=634"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/jackofalltradesmasterofsome.com\/blog\/wp-json\/wp\/v2\/categories?post=634"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/jackofalltradesmasterofsome.com\/blog\/wp-json\/wp\/v2\/tags?post=634"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}