The first process in any data warehouse project is obtaining the data into a staging environment. In the traditional data warehouse, this required an ETL process to pick up data files from a local folder or FTP, and in some cases, a direct SQL connection to source systems to then load into a dedicated staging database. In the new process we will be defining we will be using an S3 bucket to be our new staging environment. Staging data will forever live in raw file data, as any analysis or query needed against this data will be handled via tools like Athena or Elastic Map Reduce which we will cover later. Keeping this data in S3 (known as Azure Blog storage in the Microsoft world) is a cheap and convenient way to store massive amounts of data for a relatively low cost. For example, the first 50 TB is stored at a cost of $0.023 per GB. In most use cases we can assume our data requirements stay under this threshold so if we are assuming 1 TB of data files, we can ball park around $23 a month for storage or $276 a year. Pretty cheap.
Big Data for The Rest of Us. Affordable and Modern Business Intelligence Architecture – An Introduction using AWS
If you google the use cases for Big Data, you will usually find references to scenarios such as web click analytics, streaming data or even IOT sensor data, but most organizations data needs and data sources never fall into any of these categories. However, that does not mean they are not great candidates for a Modern Big Data BI solution.
The Modern Data warehouse; The low-cost solution using Big Data with HDInsight and PowerShell
Organizations have been reluctant to transition their current business intelligence solutions from the traditional data warehouse infrastructure to big data for many reasons, but there are two reasons that are false barrier to entries, cost and complexity of provisioning environments. In the post below, we will cover how to use PowerShell to commission and decommission HDInsight clusters on Azure. If used correctly, this allows organizations to cut cost down to the sub $100 dollars a month for a big data solution by creating the clusters on demand on weekends or evenings, processing all the heavy big data they may have, creating the data files required and then deleting the cluster so no charges are incurred the rest of the time. The Storage files can remain, so when you spin up the cluster again, it is if nothing ever happened to your previous data or work. The scrips below will remove the storage files as well and will need to be modified to keep that portion around. You can also move your storage to a different resource group to avoid it being removed.
Bridging the Owner to Property Manager Data Gap
One of the more common reoccurring problems seen in the real estate space is the disconnect between the data created and stored by the individual property managers and the transfer of that very data to the respective owners, operators and fund managers. In most scenarios, individual property managers are using management and accounting tools of … Read more
The Solar System A Children’s Educational Book and Lesson’s Learned Self Publishing
As my first book on an introduction to consulting get’s edited, I wanted to try my hand at writing and illustrating a children’s book which can now be found here. The experience was quite entertaining and very educational. Self-publishing allowed me to be extra creative and allowed me to get a good first book out there relatively easily. Solar System A Children’s Educational Book and Lesson’s Learned Self Publishing
Accelerating the Staging Process for your Data Warehouse
Real Estate Data Warehouse – Accelerating the Staging Process
The script below can be used to build a staging environment for any sort of industry and not just real estate related databases. The specifics of a RE Data warehouse will be covered in future blog post. It will allow you to Accelerating the Staging Process for your Data Warehouse
Creating Embedded Yardi SSRS Reports Tutorial
In previous posts we covered topics such as getting your data out of Yardi using the FTP method and the Log shipping method to facilitate building your own custom reports outside of the tool and being able to combine the data with other operational data sources. The tool comes with a comprehensive set of reports but there is always a need to create custom reports to supplement the book of reports. The tool also gives the ability to build your own SSRS reports and place them back in the tool so that they appear in the SQL Reports section for all users to access. Below we will cover the basics on Creating Embedded Yardi SSRS Reports Tutorial.
SSRS reports in Yardi require two files – a text file containing your SQL, and an RDLC containing the report definition.
If the contents of this post are outside of your wheelhouse, don’t worry, this is why I started CREXchange.io to help clients get up and off the ground with Yardi Data Integration and Advanced Power BI Custom Reports to help run your CRE business. Contact us there for more details or a demo!
Automate your Real Estate Software – Yardi ETL and Automation
Interesting in learning how to Automate your Real Estate Software – Yardi ETL and Automation. This blog post will help you drive your CRE business forward.
When running your Real Estate business, it helps to not have to think about the small things. This is where process automation can help free up time for your resources and eliminate the risk of missing tasks or checklist items when certain actions occur.
Need help with getting this solution up and running for your business? Email me at bettermentstudio5@gmail.com to discuss a low cost solution to host your reporting, analytics and integration needs!
If the contents of this post are outside of your wheelhouse, don’t worry, this is why I started CREXchange.io to help clients get up and off the ground with Yardi Data Integration and Advanced Power BI Custom Reports to help run your CRE business. Contact us there for more details or a demo!
Jack of All Trades Master of Some – An Introduction to Consulting Book
Two years ago as I reflected back on my career, I decided to begin documenting some of my stories and lessons learned to share on a blog. This exercise quickly turned into pages of a book on the topic of as Introduction to Consulting Book which is currently being edited for release sometime in 2018. Staying true to the original intention, I will begin to post sections to this blog as well. Hopefully it helps those new to consulting!
Real Estate Data Analytics Tenant Analytics – Tenant Analytics and Boosting your Cap Rate per Unit
Do you own a few rental properties or a multi-family unit? Do you know your best tenant? Is it the tenant that pays slightly lower than market rent but always pays on time? Is it the tenant that pays above market rent but seems to have some sort of maintenance request every other month? Maybe … Read more