33. Medallion Architecture and Change Data Feed

No video

33. Medallion Architecture and Change Data Feed

Рет қаралды 13,187

Жыл бұрын

Follow me on Linkedin
/ bhawna-bedi-540398102
Instagram
www.instagram....
A medallion architecture is a data design pattern used to logically organize data in a lakehouse, with the goal of incrementally and progressively improving the structure and quality of data as it flows through each layer of the architecture (from Bronze ⇒ Silver ⇒ Gold layer tables). Medallion architectures are sometimes also referred to as "multi-hop" architectures.
Change Data Feed (CDF) feature allows Delta tables to track row-level changes between versions of a Delta table. When enabled on a Delta table, the runtime records “change events” for all the data written into the table. This includes the row data along with metadata indicating whether the specified row was inserted, deleted, or updated.
Data-bricks hands on tutorials
• Databricks hands on tu...
Azure Event Hubs
• Azure Event Hubs
Azure Data Factory Interview Question
• Azure Data Factory Int...
SQL leet code Questions
• SQL Interview Question...
Azure Synapse tutorials
• Azure Synapse Analytic...
Azure Event Grid
• Event Grid
Azure Data factory CI-CD
• CI-CD in Azure Data Fa...
Azure Basics
• Azure Basics
Data Bricks interview questions
• DataBricks Interview Q...

Пікірлер: 23

@aneeshmarathe7269 Жыл бұрын

Loved your explanation. Simple and to the point 👌

@bonasiuday Жыл бұрын

Crystal clear presentation 👏

@corybeyer1 2 ай бұрын

Another amazing video, thanks!

@suresh.suthar.24 Жыл бұрын

Thank You mam for databricks series.

@atbplibrarychannel695 Жыл бұрын

Great Demo!

@sravankumar1767 Жыл бұрын

Superb explanation 👌 👏

@moughosh3640 Жыл бұрын

Crystal Clear Explanation. You have hard coded the version of the table while merging , when you schedule in a pipeline , how to make it dynamic ? is it like get max ( _change_type) and then use it in merge?

@raviv5109 Жыл бұрын

Hey thank you so much for creating this! One question - how can i always pick up the latest version of changes?

@shubhamchaturkar8324 Жыл бұрын

Hi Bhavana, great explanation for CDF. Now when update happens on a table , table_changes contains two records update_preimage and update_postimage. I am more curious about how you handle these cases while taking table into next merge. because i can see you take table name and version only.

@vipinkumarjha5587 Жыл бұрын

Hi Thanks for the video on new feature of Data bricks. I have one question , you have hardcoded the version number in code. how can we pass the latest version dynamically in merge part. Thanks in advance

@felipecastro3710 Жыл бұрын

Did you find the answer for that? The only way I can think of, would be to store a checkpoint somewhere for the latest processed _commit_version...

@priyankam3977 Жыл бұрын

@@felipecastro3710 how do you do that and I am looking for an answer for the same question.

@priyankam3977 Жыл бұрын

Great video and explanation. I have a question - In Cmd22- For Merge stmt we need to keep updating the version number from 2 to 3 and so on for each incremental load right based on the latest version. How do I do that automatically?

@vipuljain-ok9qy Жыл бұрын

Thanks Bhawna for sharing the knowledgeable video.. Is it possible to share this code through sharing Apps like GIT so it would be easy to do hands-on

@obedvaldes3680 9 ай бұрын

When you selected from the silver table on the table_changes, you need to pick the postimage record only , correct ?

@philfang1822 9 ай бұрын

At 18:08, 2nd insert of 7 rows, Six rows are on date "4/1/2021", One row on date "3/1/2021" and they are inserted, how come at 18:28, showing 7 rows inserted, but Six of them are on date "3/1/2021" and One row is on "4/1/2021" ?? I'd expect them to be the same dates as of 18:08 ? Am I missing something or I got something wrong ? Can someone explain to me please ?