Setting up the ETL_Control Database

Create the database ETLControl and the table to store the metadata for the ETL runs.

USE [ETLControl]
GO

SET ANSI_NULLS ON
GO

SET QUOTED_IDENTIFIER ON
GO

CREATE TABLE [dbo].[ETLControl](
	[Id] [int] IDENTITY(1,1) NOT NULL,
	[DatabaseName] [varchar](50) NOT NULL,
[SchemaName] [varchar](50) NOT NULL,
	[TableName] [varchar](50) NOT NULL,
	[LoadType] [varchar](50) NOT NULL
) ON [PRIMARY]
GO


Insert Into [dbo].[ETLControl]
Select 'Databasename', 'dbo', 'TableName1', 'Full'

Insert Into [dbo].[ETLControl]
Select Dataasename', 'dbo', 'TableName2', 'Full'

Setting up Azure Data Factory

1. Create a Linked Service to the SQL Database

• Create a DataSet for the ETLControl Database
  • Point to the Linked Service for SQL Server
  • Do no assign it a table name. This will be done dynamically later.

• Add a new Pipeline with the Lookup object
  • Set the source Query  “Select * From ETLControl”

• Add the For Each Loop
  • In the settings add the Dyanmic Item “@activity(‘Get-Tables’).output.value”

• Add a new data source for the SQL Souce
  • Give the SQL a parameter for TableName and SchemaName
  • Update the Table to use the variables

• Add a new data source for the DataLake Destination
  • Give the SQL a parameter for FileName
  • Update the File Path to the Dynamic content parameter

• Add a Copy Activity to the For Each Loop.
  • Set the source using the variables from the Lookup
  • Set the sink as the file name variable from look up with .csv

• Debug to run to see new files land in Data Lake with dynamic names. There should be one file for each table that was loaded. You can modify the file names to include folder names and more dynamic storage if needed.

Looping SQL Tables to Data Lake in Azure Data Factory

Setting Up Twitter

First we will want to create a Twitter Account and Handle and register for a developer account. This may take a few days for you to get approved.

1. Create a Twitter Account

• After a few more steps you will have a working Twitter Account

• Apply for a developer tool. For this we are going to build a hobbyist bot.
  • https://developer.twitter.com/en/docs/basics/developer-portal/overview

• If you have not verified a phone number, it will make you do that now.
• The next few screens will have you fill out why you want to use the portal and review and sign a developers agreement.

• You will than need to wait for your account to be confirmed

Creating an Twitter App

1. Once you app is Confirmed, you can create an app to get an API key

• Obtain your consumer keys and your access tokens, once the app is created

Create a CSV for Data

1. Create a CVS File called “post.csv” with your post message, hashtags and urls. If this is not in the same folder as your Python script this will need to be updated in the next steps

Creating a Python Application

1. Create a new python application and run the command “pip install tweepy” to install the library

pip install tweepy

• Import your excel file into a dataframe and use the sample command to select a random row. Place each of these values in their own variables.

import pandas as pd
df = pd.read_csv('post.csv')

#print (df)

df2 = df.sample()

title = df2.iloc[0]['Title']
hashtags = df2.iloc[0]['hashtags']
url = df2.iloc[0]['url']

print (title)
print (hashtags)
print (url)

• The following code can now be use to auto post your first twitter message once you replace your keys!

import tweepy

def main():
    twitter_auth_keys = { 
        "consumer_key"        : "REPLACE_THIS_WITH_YOUR_CONSUMER_KEY",
        "consumer_secret"     : "REPLACE_THIS_WITH_YOUR_CONSUMER_SECRET",
        "access_token"        : "REPLACE_THIS_WITH_YOUR_ACCESS_TOKEN",
        "access_token_secret" : "REPLACE_THIS_WITH_YOUR_ACCESS_TOKEN_SECRET"
    }
 
    auth = tweepy.OAuthHandler(
            twitter_auth_keys['consumer_key'],
            twitter_auth_keys['consumer_secret']
            )
    auth.set_access_token(
            twitter_auth_keys['access_token'],
            twitter_auth_keys['access_token_secret']
            )
    api = tweepy.API(auth)
 
    tweet = title + ' ' + hashtags + ' ' + url
    status = api.update_status(status=tweet) 
 
if __name__ == "__main__":
    main()

Interested in learning more? Check out our blog post on an Introduction to Python Class.

Automate your Twitter with API and Python – Robotic Python Automation

Looking to spruce up your home office? Here is a quick fun project you can do in Python to Create Abstract Art Automatically to print out for your home! This tutorial uses the basics of working with images, files, math and basic loops to create images you can color in however you like to match the style of your home.

Check out my free Udemy Classes here to learn Python for free! And be sure to check out my books for additional free educational material. Other materials can be found here.

• Create a new Jupyter Notebook in the following location “C:\Users\yourusername\Documents\Python\Training\”

• Create a new folder on your C Drive “C:\Users\yourusername\Documents\Python\Training\art” and create a empty file called “template.png” with a transparent background.

• Install pillow so we can work with images in Python

pip install pillow

• We than use numpy and matplot lib to generate a random shape with the following code. Change some of the constants below to experiment with generating different shapes. The loop (i) can also be changes to generate more or less shapes. This will create 25 shape in Python which you can see in Juptyer and also save them to the art folder.

from matplotlib.path import Path
import matplotlib.patches as patches
import numpy as np
from matplotlib import pyplot as plt

i=1

while i<20: 
    # UPDATE THESE VARIABLES TO CHANGE THE TYPES OF SHAPES
    n = 4 # Number of possibly sharp edges
    r = 0.7 # magnitude of the perturbation from the unit circle, 
    # should be between 0 and 1
    N = n*3+1 # number of points in the Path
    # There is the initial point and 3 points per cubic bezier curve. Thus, the curve will only pass though n points, which will be the sharp edges, the other 2 modify the shape of the bezier curve

    angles = np.linspace(0,2*np.pi,N)
    codes = np.full(N,Path.CURVE4)
    codes[0] = Path.MOVETO

    verts = np.stack((np.cos(angles),np.sin(angles))).T*(2*r*np.random.random(N)+1-r)[:,None]
    verts[-1,:] = verts[0,:] # Using this instad of Path.CLOSEPOLY avoids an innecessary straight line
    path = Path(verts, codes)

    fig = plt.figure()
    ax = fig.add_subplot(111)
    
    patch = patches.PathPatch(path, facecolor='none', lw=0.5)
    ax.add_patch(patch)

    ax.set_xlim(np.min(verts)*1.1, np.max(verts)*1.1)
    ax.set_ylim(np.min(verts)*1.1, np.max(verts)*1.1)
    ax.axis('off') # removes the axis to leave only the shape
    
    plt.savefig('art/foo'+str(i)+'.png', transparent=True, dpi=300)
    
    i=i+1

• In our last step all we need to do now is loop through each of the files we created and pasting them into the template image. We use the random function to paste them into random spots on the new image and then save the file as “workofart.png”. This create a cool new pattern on a transparent background!

from PIL import Image, ImageDraw
import numpy as np
import os
import random

i=1
#os.remove("art/workofart.png")
background = Image.open("art/template.png")


while i<20: 
    foreground = Image.open("art/foo"+str(i)+".png")
    background.paste(foreground, (random.randint(-1000,1000), random.randint(-1000,1000)), foreground)
    background.save("art/workofart.png")
    i=i+1
    

background.close()
foreground.close()  

The last step is have some fun. Use your favorite paint colors. Once completed, take over to your local print shop and put in a frame above your desk! Take a look at the artwork I created below.

Check out my free Udemy Classes here to learn Python for free! And be sure to check out my books for additional free educational material. Other materials can be found here.

Robotic Python Automation – Python to Create Abstract Art Automatically

Sometimes we have folders where files are saved that get out of control. Sometimes we have Excel documents or Work files that pile up and we need an efficient way to sift and sort through them and either archive files or delete them. Using Python, we can build a script that automatically completes two tasks.

1. Looks for files of a certain name and archives them
2. Looks for files older than a certain data and deletes them

Setup

1. Create a new Jupyter Notebook and name it Document_Automation

• Create a new folder on your C Drive “C:\Python\Automation” and seed it with 10 empty files that follow the naming conventin file_YYYYMMDD and log_YYYYMMDD. Create two folders called “Files” and “Logs”

Setup

• The following command will list out all text files located in the folder you are searching for and place the date in to a list called “mylist”. The “*” is a wildcard that allows you to search for all character combinations that end with “.txt”. This can be updated to look for other file types

import glob
mylist = glob.glob("C:/Python/Automation/*.txt")
print(mylist)

• Next we can try to create our folders where we will organize our files. Since we already created them, the try/catch will fail but if you had deleted them you will see them reappear. The OS and shutil need to be imported for these commands to work

import os, shutil
try: 
    os.mkdir("C:/Python/Automation/Files")
    os.mkdir("C:/Python/Automation/Files")
except OSError:
     print ("Folder Creation Failed")
else:
    print ("Folder Creation Success")

• Start with a loop to go through your list

for i in mylist: 
     print(i)

• Lets update the look to search for any “log” first. This can be done with a string startswith() command or using the “in” command with an If statement. Both work in the case but be careful if your file naming rules can change. We will use the latter example for the rest of the tutorial.

for i in mylist: 
    if (i.startswith('C:/Python/Automation\log_')):
        print(i)

for i in mylist: 
    if ("log" in i):
        print(i)

• Now move the files found in the search to the destination folder using the Move command. We need to specify the source location and destination location so we use Replace string manipulation to create both of those strings to pass to the function. Once run, you will see the files moved to the respective folders.

for i in mylist: 
    if ("log" in i):
        #print(i)
        source = i
        destination = i.replace('C:/Python/Automation', 'C:/Python/Automation/Logs') 
        shutil.move(source, destination)

• We can now do the same with the files titled “Files”

for i in mylist: 
    if ("file" in i):
        #print(i)
        source = i
        destination = i.replace('C:/Python/Automation', 'C:/Python/Automation/Files') 
        shutil.move(source, destination)

Cleaning out old Files

• Finally, we want to delete any old files. We can do this using the remove and time functions in Python to identify the last modified time of the file. In this sample 1800 is 30 minutes in seconds. This can be adjusted to make the window longer or a few days. We need to import time and sys to be able to the correct commands. Uncomment the If statement to use parameters on the time duration.

import os, time, sys

path = r'C:/Python/Automation/Logs'

now = time.time()

for f in os.listdir(path):
    #if os.stat(os.path.join(path,f)).st_mtime < now - 7 * 86400:
    if os.stat(os.path.join(path,f)).st_mtime < now:
        print("Removing File")
        os.remove(os.path.join(path,f))

Check out my these Udemy Classes below to get started in Python

And Other articles here below.

Robotic Python Automation Files Cleaning Sorting and Archiving