Side Note: Want to learn SQL or Python for free. In less then 10 minutes a day and less than an hour total? Signup for my free classes delivered daily right to your email inbox for free!

Now back to the article…

1. Log into Salesforce as an admin account and navigate to settings.

• Reset your security token on the left hand side menu under “My Personal Information”. This will email you a token that will be used in later steps.

• Navigate to “Setup”

• Navigate to “Setup -> Permission Sets -> (name) -> System permissions” and edit the “Security Center Integration User” and then the “System Permissions”

• Verify that Enable “API Enabled”

Create a Linked Service to Salesforce in Azure Data Factory

1. Create a new Linked Service for Azure
2. Obtain your environment name from Salesforce and log in with your credentials and service key from the previous steps. You should be able to now test your connection to verify it works. you will need the security token that was emailed to you as well in this step.

Create a Azure Data Factory Dataset to Salesforce

1. Create a new dataset in ADF and Select Salesforce

• Point to the linked Service you created for Salesforce in the previous step. Select the table you wish to interact with. In this case, it will be the Contact table. Hit preview data to ensure the connection is working and set up correctly. Name the dataset and Save to complete.

Create a pipeline to move Data

The last step is the easy part. Create copy commands in the Pipelines to move data in and out of Salesforce. This can be used to update contacts, accounts, leads and other values automatically form other systems and processes.

Copy data to and from Salesforce with Azure Data Factory

Get Salesforce here

Get Azure here

Enjoying Learning about Azure and Azure Data Factory? Check out some of my online classes here or how to set up a run time to move data on-prem.

1. Enabling CDC
2. Setting up Audit Tables
3. Provisioning Azure Data Factory
4. Provisioning Azure Blog Storage
5. Create Data Source Connection in ADF
6. Create Incremental Pipeline in ADF
7. Create a Parameter Driven Pipeline
8. Create a Rolling Trigger

This series uses the Adventureworks database. For more information on how to get that set up see my Youtube video for Downloading and Restoring the database.

Enabling CDC on SQL Server

1. First, enable Change Data Capture at the Database and Table Level using the following scripts. More information is available on the Microsoft Site. If you are not using SQL Server or a tool that has organic CDC build it, a similar process can be hand built a read only view by leveraging timestamps on last created date and last updated date.

/********Code**********/

sys.sp_cdc_enable_db

EXECUTE sys.sp_cdc_enable_table

    @source_schema = N’dbo’

  , @source_name = N’DimProduct’

  , @role_name = N’cdc_Admin’;

GO

• This step will enable CDC on the database as well as add it to the table “DimProduct”. SQL Agent must be running as two jobs are created during this process, one to load the table and one to clean it out. This is a new process that will consume resources to be sure to do your research before turning this on for all tables.

• Once the CDC is enabled, SQL Server will automatically create the following tables “schema_tablename_CT” in the System folder section. This table will now automatically track all data changes that occur on that table. You can reference the _$operationcode to determine what change occurred with the legend below. If you wish to capture changes to only certain fields, see the Microsoft documentation on CDC to see how that can be set. If you are handwriting your SQL, this can also be programmed in when building your staging query.

1 = delete

2 = insert

3 = update (captured column values are those before the update operation). This value applies only when the row filter option ‘all update old’ is specified.

4 = update (captured column values are those after the update operation)

• Now once you add, edit or delete a record, you should be able to find it in the new CDC table. Next, we will look at scanning this table and turning the data to JSON to send to an Event Hub! Run the code below to insert and update into the DimProduct table.

Insert

INSERT INTO [dbo].[DimProduct]

           ([ProductAlternateKey]

           ,[ProductSubcategoryKey]

           ,[WeightUnitMeasureCode]

           ,[SizeUnitMeasureCode]

           ,[EnglishProductName]

           ,[SpanishProductName]

           ,[FrenchProductName]

           ,[StandardCost]

           ,[FinishedGoodsFlag]

           ,[Color]

           ,[SafetyStockLevel]

           ,[ReorderPoint]

           ,[ListPrice]

           ,[Size]

           ,[SizeRange]

           ,[Weight]

           ,[DaysToManufacture]

           ,[ProductLine]

           ,[DealerPrice]

           ,[Class]

           ,[Style]

           ,[ModelName]

           ,[LargePhoto]

           ,[EnglishDescription]

           ,[FrenchDescription]

           ,[ChineseDescription]

           ,[ArabicDescription]

           ,[HebrewDescription]

           ,[ThaiDescription]

           ,[GermanDescription]

           ,[JapaneseDescription]

           ,[TurkishDescription]

           ,[StartDate]

           ,[EndDate]

           ,[Status])

     VALUES

           (‘VV-2903’

           ,NULL

           ,NULL

           ,NULL

           ,’Test Product’

           ,’Test Product’

           ,’Test Product’

           ,0

           ,0

           ,’Black’

           ,NULL

           ,NULL

           ,NULL

           ,NULL

           ,NULL

           ,NULL

           ,NULL

           ,NULL

           ,NULL

           ,NULL

           ,NULL

           ,NULL

           ,NULL

           ,NULL

           ,NULL

           ,NULL

           ,NULL

           ,NULL

           ,NULL

           ,NULL

           ,NULL

           ,NULL

           ,NULL

           ,NULL

           ,NULL)

GO

Update

UPDATE [dbo].[DimProduct]

SET FinishedGoodsFlag = 1

WHERE [ProductAlternateKey] = ‘VV-2903’

• Now you can query the table “cdc.dbo_DimProduct_CT” to see the changes that were recorded.

Streaming ETL with Azure Data Factory and CDC – Enabling CDC

The difference between a good looking Power BI report and a great looking Power BI report is just a short extra mile using a custom background template. It is easy to create or download free custom backgrounds to make your Power BI reports pop!

Using basic best practices in material design, it is pretty easy to create a series of beautiful backgrounds for your reports to give them a professional look. If you are not a designer, you can download 10 free templates from my website as well.

Here is one basic report as created in the basic designer.

And here is one using a custom background.

Download 10 free templates from here or reach out of you need help!

1. We will first need to set up the necessary extensions. Click on the icon below located on the right-hand side and install the following extensions.
   a. Flutter
   b. Dart
2. Create a new empty folder and then open it in Visual Studio Code
3. Create a new flutter project by opening Terminal and Navigate to the folder you created. Here is the starter location for my No0Days app.

1. Run the command “flutter create yourappname” and replace yourappname with the correct name to get it created. The terminal will autorun flutter doctor after as a health check. Inside Visual Studio Code, you will now see your new project. You will need to be connected to the internet for this step.

1. Go ahead and start your device emulator if you have not already by typing “open -a simulator” in your terminal window.

1. Navigate to inside the root folder by going one layer down from where you are in terminal. “cd no0days”, and then run “flutter run” to run your first app. After a few seconds of installing the app, you should now have your first up and running application! The cool thing about Flutter is that it allows you to hot reload the application while developing so you can see your changes instantly. Just hit “r” in the terminal to reload and voila, your app changes appear!

1. Download and install all the necessary tools and applications to your computer.
   a. Android Studio for Mac
   b. Xcode from the App store
   c. Flutter from Google (https://flutter.io). Remember where you have unzipped the Flutter files. I placed them under my “documents/src/flutter” and will use that path for the rest of the tutorial.
   d. Homebrew (brew.sh)
2. Launch Terminal and navigate to the folder of where you unzipped the files in step 1c.
3. Type “pwd” to get the present working directory. For me it appeared as “/Users/vimalvachhani/documents/src/flutter”

1. Type in the command “export PATH=”/Users/vimalvachhani/documents/src/flutter/bin:$PATH”. But this only creates a tempory entry. Let’s go create a permanent entry instead so you can skip this step other time.
2. Navigate to “.bash_profile” by going to your root directory. Type “cd” over and over until your at the root folder. They type “ls -la” to show the files. We are looking for a file called “.bash_profile” but in my case, it did not exist. If it is not, use a text editor to create the file to get started using “vim” or other tools.

1. In the file now copy and paste the command “export PATH=”/Users/vimalvachhani/documents/src/flutter/bin:$PATH” and save the file.
2. Back in the terminal, type the words “flut” and hit the tab key. If it auto-populates to “flutter” we are in good shape. If not check the above steps to make sure you got everything correct. If you ever move your flutter directory, you will need to update this path.
3. Verify Xcode and Android studio are set up from step one and in a terminal run the command “Flutter Doctor”. The issues you get here will vary a bit but the tool does a great job of listing all the commands you will need to run to fix them pretty easily.

That should be it! You will need to set up a simulator but in the next tutorial, we will walk through setting up VS Code to build your apps!