Delta Live Tables: Building Reliable ETL Pipelines with Azure Databricks

No video

Delta Live Tables: Building Reliable ETL Pipelines with Azure Databricks

Рет қаралды 34,831

Күн бұрын

In this session, we will see how to use Delta Live Tables to build fast, reliable, scalable, and declarative ETL pipelines on Azure Databricks platform.
Speaker: Mohit Batra SQLbits.com/sp...
SQLbits.com/Ses...
Tags: Azure,Spark,Data Lake,Big data analytics,Developing,Data Bricks,Big Data & Data Engineering,Data Loading Patterns & Techniques

Пікірлер: 45

@supriyasharma9517 11 күн бұрын

Great video and easy explanation. I hope you come up with a series on step by step on Databricks for beginners like me who are finding to difficult / struggling to make the switch. Thanks for your efforts

@samanthamccarthy9765 6 ай бұрын

Awesome thanks so much . this is really useful for me as a Data Architect . much is expected from us with all the varying technology

@amadoumaliki 6 ай бұрын

As usual! Mahit wonderful!

@Rangapetluri 2 ай бұрын

Wonderful session. Sensible questions asked. Cool

@artus198 9 ай бұрын

I sometimes feel, the good old ETL tools like SSIS , Informatica were easier to deal with ! 😄 (I am a seasoned on premise SQL developer, transitioning into the Azure world slowly).

@menezesnatalia 9 ай бұрын

Nice tutorial. Thanks for sharing. 👍

@ananyanayak7509 Жыл бұрын

Well explained with so much clarity. Thanks 😊

@SQLBits Жыл бұрын

Our pleasure 😊

@ADFTrainer 9 ай бұрын

@@SQLBits Can you provide code. Thanks in advance..

@MichaelEFerry 10 ай бұрын

Great presentation.

@SQLBits 10 ай бұрын

Thanks for watching :)

@germanareta7267 Жыл бұрын

Great video, thanks.

@starmscloud Жыл бұрын

Learned a Lot from this . Thank You for this video !

@SQLBits Жыл бұрын

Glad it was helpful!

@pankajjagdale2005 10 ай бұрын

crystal clear explanation thank you so much can you provide that notebook ?

@Databricks 11 ай бұрын

Nice video🤩

@SQLBits 10 ай бұрын

🥳

@priyankpant2262 4 ай бұрын

Great video ! Can you share the github location of the files used ?

@anantababa 5 ай бұрын

Awesome training, can you please share the data file, i want to try it.

@walter_ullon 3 ай бұрын

Great stuff, thank you!

@trgalan6685 Жыл бұрын

Great presentation. No example code. What's zero times zero?

@prashanthmally5765 3 ай бұрын

Thanks SQLBits. Question: Can we create a "View" on Gold Layer instead having "Live Table" ?

@srinubathina7191 Жыл бұрын

Wow super stuff thank you sir

@SQLBits Жыл бұрын

Glad you liked it!

@supriyasharma9517 9 күн бұрын

can you please provide code for this?

@user-cg6yw8ei6j 7 ай бұрын

For complex rule based transformations how we can leverage it?

@ashwenkumar 6 ай бұрын

Does delta live tables in all the layers has filesystem linked to it as like in hive or Databricks ?

@MohitSharma-vt8li Жыл бұрын

Can you please provide us the notebook DBC file or ipynb.. By the way great session, Thanks

@SQLBits Жыл бұрын

Hi Mohit, you can find all resources shared by the speaker here: events.sqlbits.com/2023/agenda You just need to find the session you're looking for and if they have supplied us with their notes etc, you will see it there once you click on it!

@MohitSharma-vt8li Жыл бұрын

@@SQLBits thanks so much

@freetrainingvideos Ай бұрын

Very well explained, Thanks

@user-cg6yw8ei6j 7 ай бұрын

Is there any way to load new files sequentially if bunch of files arrived at a time?

@olegkazanskyi9752 Жыл бұрын

Is there a video on how data is pulled from the original source, like a remote SQL/noSQL server, or some API? I wonder how data is getting to the data lake? I assume this first extraction should be a bronze layer.

@lostfrequency89 5 ай бұрын

Can we create dependency between two notebooks?

@guddu11000 6 ай бұрын

shoud have showed us how to trobleshoot or debug

@thinkbeyond18 Жыл бұрын

I have a general doubt in autoloader . does autoloader required to run in a job or notebook triggering manually .Or no need to touch anything once we written the code as when as the file arrives it will run automatically and processed the files.

@Databricks 11 ай бұрын

Trigger your notebook that contains your DLT + Auto Loader code with Databricks Workflows. You can trigger it using a schedule, a file arrival, or choose to run the job continuously. It doesn't matter how you trigger the job. Auto Loader will only process each file once.

@ADFTrainer 9 ай бұрын

pls provide code links

@tratkotratkov126 4 ай бұрын

Hm … Where in these pipelines you have specified that nature of the created/maintained entity - bronze, silver or gold other then the name of the object itself. Also where these LIVE tables are exactly stored - from your demonstration it appear they all live in the same schema / database while in real live the bronze, silver and gold entities have designated catalogs and schemas.

@TheDataArchitect 8 ай бұрын

I don't get the usage of VIEWS between Bronze and Silver tables.

@TheDataArchitect 8 ай бұрын

Anyone?

@SQLBits 7 ай бұрын

Hi Shzyincu, you can get in touch with the speakers who taught this video via LinkedIn and Twitter if you have any questions!

@richardslaughter4245 4 ай бұрын

My understanding (as an "also figuring out data bricks" newb: * View: Because the difference between bronze and silver in this instance is very small (no granularity changes, no joins, no heavy calculations, just one validation constraint), it doesn't really make sense to make another copy of the table when the view would be just as performant in this case. * "Live" view: I think maybe this is required because the pipeline needs it to be a live view to properly calculate pipeline dependencies Hopefully that understanding is correct, or others will correct me :) My follow up question would be: As I think about that validation constraint, it really seems like in this case it seems functionally identical to just applying a filter on the view. Is that correct? If so, is the reason to use the validation constraint rather than a filter, mostly to keep code consistency between live tables and live views?

@anilkumarm2943 Ай бұрын

You don't materialize as new tables evertime, We sometimes materialize it as views. So minor transformations like changing the type of the field etc.