Tag: Azure
-
Setting up Azure Data Factory Integration Runtime for On-Prem Connections
If you need to connect your Azure Data Factory to an on-premise SQL server you will need help Setting up Azure Data Factory Integration Runtime for On-Prem Connections Set up On-Prem Integration Run Time Click on Author & Monitor from Azure Data Factory Create two new linked services Linked Service to Database. Since we are…
-
Auto Spin up and Down Azure Virtual Machines (VM)
One of the best perks of moving your infrastructure to the cloud is the ability to only pay for the resources you use. When using Virtual Machines that usage can be even further limited to servers that stay on only when you need them (business hours or nightly data runs) and then shut down to…
-
Streaming ETL using CDC and Azure Event Hub. A Modern Data Architecture.
In Modern Data architecture, As Data Warehouses have gotten bigger and faster, and as big data technology has allowed us to store vast amounts of data it is still strange to me that most data warehouse refresh processes found in the wild are still some form of batch processing. Even Hive queries against massive Hadoop…
-
The Modern Data Warehouse; Azure Data Lake and U-SQL to combine data
The modern data warehouse will need to use Azure Data Lake and U-SQL to combine data. Begin by navigating to your Azure Portal and searching for the Data Lake Analytics Resource. Let’s start by creating a new Data Lake. Don’t worry, this service only charges on data in and out, not just remaining on like…
-
The Modern Data Warehouse; Running Hive Queries in Visual Studio to combine data
In previous posts we have looked at storing data files to blob storage and using PowerShell to spin up an HDInsight Hadoop cluster. We have also installed some basic software that will help us get going once the services are provisioned. Now that the basics are ready, it is time to process some of that…
-
Setting up tools to work with HDInsights and run Hive Queries – Azure Data Lake Tools and Azure Storage Browser
Two tools that are going to make life a bit simpler if you are going to be working with HDInsights and Azure blog storage are “Azure Data Lake and Stream Analytic Tools for Visual Studio” and Azure Storage Browser. Azure Data Lake and Stream Analytic Tools for Visual Studio To run Hive Queries, you’re going…