Azure Data LakeHouse in an Hour Virtual Workshop

No video

Azure Data LakeHouse in an Hour Virtual Workshop

Рет қаралды 5,661

Күн бұрын

In the Spirit of popular "Dashboard in a Day" workshops, this virtual workshop walks you through both the basic concepts and a hands-on build of an Azure Data LakeHouse using public CDC National Notifiable Diseases Surveillance System (NNDSS) data. If you don't write code, fear not! There is minimal code in this workshop and a link to a document where you can cut & paste a few snippets of code is included below. You can follow along with the video to build out your own Azure Data LakeHouse using Azure Data Lake and Azure Data Factory. Hopefully after watching this video you'll gain a basic understanding of the LakeHouse architecture, reasons it is valuable, and how it can be applied. If you follow along and build your own Data LakeHouse, you'll also have a functional Azure Data LakeHouse with weekly updates of CDC NNDSS data.
Document with "Cut and Paste" Code Snippets - github.com/gre...
CHAPTERS
0:00 Intro (Get an Azure Account link: azure.microsof... )
3:06 Overview of CDC National Notifiable Diseases Surveillance System (NNDSS) data (www.cdc.gov/nn... )
4:04 Overview of Azure Data LakeHouse Architecture
6:34 Step One - Create Resource Group, Azure Data Lake, and Azure Data Factory
8:50 Step Two - Rapid Prototype of the Data Model
12:43 Step Three - Define the Datasets in Azure Data Factory
23:21 Step Four - Populate the Bronze Layer of the Azure Data LakeHouse
26:12 Step Five - Populate the Silver Layer of the Azure Data LakeHouse
42:31 Step Six - Populate the Gold Layer of the Azure Data LakeHouse
50:53 Step Seven - Automate the Weekly Update and Store Historical Snapshots
1:03:53 Cost Review
1:04:40 Outro

Пікірлер: 7

@jeremyloscheider833 2 жыл бұрын

I don't disagree with bronze/silver/gold in the file path. The suggestion to associate with a project or domain within the lakehouse file path makes sense. I will note that bronze/silver/gold can have unintended meanings so using raw/processed/enriched may be more clear.

@insightsoutliers 2 жыл бұрын

Jeremy, I agree and would probably name it differently if I started over. From my perspective there is quite a bit of freedom to organize different steps of a project based upon organizational norms. Naming conventions, User access per Security requirements, Organizing Multiple Data Lakes, hierarchical organization patterns in Data Lakes, CICD patterns, etc can all be different. One piece of advice I'd hold firm is to standardize on an architectural pattern that can scale across collaborative teams and projects to prevent a data lake from becoming a data swamp.

@apocalipto91 Жыл бұрын

Awesome!!

@sunnysaneeth6693 4 ай бұрын

Why did you use date.csv file in this ?

@insightsoutliers 4 ай бұрын

At the time it was a simple way to add a Date table to the Gold layer. It also shows that you can strategically unite data from different sources with a Lakehouse architecture. I'd probably do it differently today with Fabric.