Master Databricks and Apache Spark Step by Step: Lesson 1

Master Databricks and Apache Spark Step by Step: Lesson 1 - Introduction

Рет қаралды 113,558

Күн бұрын

In this first lesson, you learn about scale-up vs. scale-out, Databricks, and Apache Spark. This video lays the foundation of the series by explaining what Apache Spark and Databricks are. The series will take you from Padawan to Jedi Knight! Join me!
Join my Patreon Community
www.patreon.com/bePatron?u=63...
Twitter: @BryanCafferky
Slides and Other Content when Applicable available at:
github.com/bcafferky/shared/t...

Пікірлер: 89

@faisala1037 2 жыл бұрын

There's a ton of videos ( one for every keyword) on KZfaq on this subject. Most fails to deliver any useful knowledge, others are too narrow and/or incomprehensible. I'm so glad to have found this series. Your teaching style took me back to my college classes. Fairly detailed and well explained. So a big thanks to you for it Bryan 👍.

@BryanCafferky 2 жыл бұрын

Thanks, Faisal. If you follow the entire series, you will get a solid foundation.

@mansah707 Ай бұрын

@@BryanCafferky I intend to go through this whole stuff.. Lesson 0 and 1 completed... onto lesson 2

@animeshmohanty5052 Жыл бұрын

You are awesome! There's hardly any other material which is as clear and condensed. Thank you for creating this video🙏

@abhinavkashyapv 2 жыл бұрын

This video clearly explains the concepts around apache spark, databricks and the various offerings. Wonderful explanation thanks a ton 👏👍

@dhwanik02 7 ай бұрын

This is one of the best and clearest explanations about Spark and Databricks on the internet.

@user-fp3zc6kw4r 2 ай бұрын

I just found this while dropping a bunch of other ones, and yes I can confirm, indeed this is the best one so far

@boubeniamohamed236 10 ай бұрын

Definetly the best serie for learning databricks

@alexandermedina4950 Жыл бұрын

Great content, thank you for doing this general and historic view, sometimes it is necessary to understand the details.

@naomilago Жыл бұрын

O M G I found what I was looking for. I've started working at Nestlé as a Data Science Analyst and I'm searching for a good playlist of Databricks and Spark to have a deeper understanding on this subject but you're the one that matched my way to learn and have lectures. A huge big thanks to you 🌟

@BryanCafferky Жыл бұрын

Thanks so much! It is really great to hear feedback like that! Glad it helps you.

@MarkFreedmanNY 7 ай бұрын

Finally, a Databricks KZfaq series that makes sense! I'm using DB with AWS, but this all pertains. Thanks!

@BryanCafferky 7 ай бұрын

You're welcome!

@samirks27 7 күн бұрын

Thanks Bryan for wonderful video, you kept me engaged and attentive through out of the video. Your explanation very crystal clear and one of the best on the internet. Thanks and god bless you healthy and energetic.

@BryanCafferky 4 күн бұрын

Thank you!

@user-hw2ls4tn5b 4 ай бұрын

Spot On! I really liked how you transitioned from the broader umbrella of Hadoop> spark> Databricks.. Great job Bryan!...

@BryanCafferky 4 ай бұрын

Thank You!

@mandarkulkarni7675 9 ай бұрын

probably the first video that describes the difference between spark and databricks so cleanly and also the different components of spark with regards to where they are placed in the whole data engineering ecosystem .... Thanks a lot ...!!!

@BryanCafferky 9 ай бұрын

You're welcome!

@marvhan888 4 ай бұрын

yeah agree, so cleanly.

@hemalpbhatt 2 күн бұрын

Love your explanation! It is so easy to understand

@dataoil8416 Жыл бұрын

Exactly what I was looking for !!! your best teacher is your last mistake! proved!

@bibinkunjumon Жыл бұрын

This is my 3 Rd teacher. You explained all well from an experienced person. I thought first what this old man gonna speak...now end up touching ur feet. Well done Bibin from India,Kerala

@ash2ucool 10 ай бұрын

Thank you, Thank you, Thank you for explaining it in the simplest way possible. At last I was able to understand what are Hadoop, Spark and Databricks, and what actually they do.

@BryanCafferky 10 ай бұрын

So glad to hear that. It's why I do this channel. Thanks

@fartknockerR17 11 ай бұрын

Hi Bryan. I'm just starting out on this topic. I knew you were the guy for me when you dropped some Klingon! Thanks for your work.

@datoalavista581 Жыл бұрын

Thank you Professor Bryan !

@alokhom Ай бұрын

your video has decluttered me a lot. Now am going to make a hdfs on my k8s cluster and spark operator

@mansah707 Ай бұрын

I have never seen such a straightforward, clear , concise explanation on this concept. till date, i have tried to understand Apache Spark and Databricks... but i've always had some convoluted understanding of them. thank you for much for this video.. it really helped me understand where things stand now.

@BryanCafferky 29 күн бұрын

Thanks. Glad the videos are helpful.

@voliteon 9 ай бұрын

Thanks for your videos Bryan - nice work. Really good amount of information clearly explained.

@BryanCafferky 8 ай бұрын

You're welcome! Thanks for watching.

@samanthamccarthy9765 7 ай бұрын

thanks really good summary of all these languages and how they came about .

@KhalilJolibois 2 жыл бұрын

thanks for these videos i'm finishing up the data camp data engineer track and then jumping in on these

@BryanCafferky 2 жыл бұрын

Great!

@andreaceribelli9705 Ай бұрын

Incredible quality, thanks!

@brenthackers132 11 ай бұрын

Guy has two left sides and still manages to make sense. Inspiring. :)

@anandchandrashekhar2933 2 жыл бұрын

Great start to the series. Thank you!

@BryanCafferky 2 жыл бұрын

YW!

@amataratsu006-xs6hv 4 ай бұрын

Sir thank you so much! You match my learning style and you have a clear voice

@BryanCafferky 4 ай бұрын

Thanks. Glad the videos are helpful!

@Hamromerochannel 2 ай бұрын

I tried to do data bricks academy and I got lost. Thanks to channel, I understand every nook and crannies. Thumbs up Brian!!

@BryanCafferky 2 ай бұрын

Thank you! Glad my videos are helping you.

@G47_Code 2 жыл бұрын

Thank you Brian so much for the wonderful contents!!!

@BryanCafferky 2 жыл бұрын

YW. Glad it is helpful.

@revidenver5142 Жыл бұрын

The Best explanation, thank you

@MeridiusMaximus 2 жыл бұрын

such a clean explanation. Thank you!

@BryanCafferky 2 жыл бұрын

@arturrizzato1034 3 ай бұрын

A very good class, especially for a Databricks virgin like me.

@lucassaito1791 2 жыл бұрын

Outstanding content!

@davidk7212 Ай бұрын

Zank you sir for zis tutorial. It is most very velcome.

@bananaboydan3642 7 ай бұрын

This is an amazing video

@srajv01 11 күн бұрын

Clingon !! That's when I subscribed 😅

@anmolchoudhary3982 2 жыл бұрын

ohh man such a detailed and superbly structured content.... I wish I could take you out for beers sometime :)

@BryanCafferky 2 жыл бұрын

Thanks. I appreciate the kind words. It's great to know my work is helpful.

@scxry5597 5 ай бұрын

Thank you so much for your videos, i have been looking for this

@BryanCafferky 5 ай бұрын

You're welcome!

@JCArtuso 2 жыл бұрын

Great! Let's go!

@BillusTinnus Жыл бұрын

Fantastic video! Really well done, thank you

@BryanCafferky Жыл бұрын

Thank you! Glad they help.

@mehmetkaya4330 Жыл бұрын

I would double that! So concise yet comprehensive overview! Thank you so much!

@BryanCafferky Жыл бұрын

@@mehmetkaya4330 Thanks!

@user-qh5qo2tr7l Жыл бұрын

Thank you very much, it was very interesting and helpful

@BryanCafferky Жыл бұрын

You're welcome!

@gustavonavesdesouza759 4 ай бұрын

Thanks for that

@Navinneroth 2 жыл бұрын

Brilliant analogy sir .. phone books example.. for distributed compute too good.

@youssefloukili1785 Жыл бұрын

thanks

@sehaj778 2 жыл бұрын

Hi Bryan, I'm currently learning Data science on GCP as a beginner. I'm just scratching the surface about learning GCP tools/platform. I wanted to learn Spark and that is why I'm here. Would learning Spark and Databricks in a 'Microsoft Azure platform' be a right idea at this time given I'm focusing on GCP ? Thanks for making this course though, I see so much content here and I'm still on the first video!

@BryanCafferky 2 жыл бұрын

Databricks is a service owned by the company Databricks that is available on AWS, Azure, and GCP. It should be the same on any of these platforms with the only differences being how cloud-specific resources are called or integrated, i.e Azure Synapse vs. Google's BigQuery. You should be fine using Databricks on GCP but let me know if you find significant differences. Make sense?

@ThEHaCkeR1529 9 ай бұрын

Thanks a lot!

@BryanCafferky 9 ай бұрын

You're welcome!

@sivachagaleti6614 Жыл бұрын

Awesome

@0yustas0 Жыл бұрын

Thank you. Looks like, Kubernetes missed in the RM section. About interactive queries... Why beeline isn't interactive when you compare Hadoop and Spark? :) if you'll say that it's slow, it isn't. If you use Hive on Tez. :)

@ishaqkhan8653 2 ай бұрын

Hey Bryan, thank you for the excellent video. it put my mind at ease. I have seen that you have used Azure Databricks going forward. However my organization stores data on s3 and works predominantly in databricks platform itself. I was wondering if the knowledge you have shared will work good in direct databricks platform. I am a complete new beginner in this field, so apology for any silly questions

@BryanCafferky 2 ай бұрын

Hi Ishaq, Databricks is a complete self contained service available on AWS, Azure, and GCP. It should work the same on all three with the only differences being how it integrates with the cloud specific back end services like s3. Also, Azure integrates Databricks in a way that eliminates the need for the customer to have an agreement with Databricks and Microsoft. It appears as if it were an Azure service. I think AWS requires customers to license with Databricks and AWS when they set it up. So yes, overall, all the Databricks and Spark code and services should be the same on all 3 cloud platforms. Make sense?

@shomero8334 Жыл бұрын

Thank you, man! I was lost at first, I needed your Tutorial so so so so much!!

@BryanCafferky Жыл бұрын

Glad it helped! I understand. It is a lot to learn.

@carlosramirez-pf1zq Жыл бұрын

thank you for your explanation about spark is ,Its confuse at firts sigh are these technologies for someone that never used .

@BryanCafferky Жыл бұрын

You're welcome!

@ROHITCHAUHAN-lu6jn 2 жыл бұрын

How to drop cached data which was cached using delta cache into local storage ? I couldn't find a proper command.

@BryanCafferky 2 жыл бұрын

That's a bit beyond the content of this video.

@rydmerlin 2 жыл бұрын

Is your book available in epub format?

@jamesschoi87 Жыл бұрын

28:10 You couldn't install external libraries with open source spark?

@BryanCafferky Жыл бұрын

You can but you can define libraries for a cluster and Databricks will automatically re-install them ever time the cluster starts. You can even define libraries you want installed on every cluster if you like. Spark does not support cluster stop and start. You have too delete and re-create clusters if you want to stop paying for them. When you create a cluster, you have do do some work to install the libraries you want.

@erkansirin6849 2 жыл бұрын

Where's Kubernetes as cluster manager?

@artus198 8 ай бұрын

In general , what I notice is , compared to the past, they are over-complicating everything, especially that whole Azure thing is unnecessarily complex. At least on-premise was never this much work !

@BryanCafferky 8 ай бұрын

No. I disagree there. In fact, the point is that Cloud based Databricks is tons easier to use and provides much better tools than using open source Spark on prem. Not sure what you are looking at. Thanks for your comment.

@artus198 8 ай бұрын

@@BryanCafferky Eg: In Databricks , If I want to access dbfs files in another resource group - you have to create a "scope', get access to a vault secret, use the scope to mount that dbfs in your workspace hive metastore, write a script to mount, write a script to create a temp view and read the data from that delta table. In SQL Server: I can share connection string user/password with somebody else, they can connect to the database from SQL Management studio, enter the details and run as many queries as they want on that database, joining multiple tables etc etc.