I’m so glad I found this video, I was just sitting here with 60 million gigabytes and was figuring out what joins to use so this was perfect timing.

Can't wait to build hyperscale pipelines for my startup with 0 users

What I absolutely love about your videos is that as a beginner in the data engineering field, you often talk about things that I had no conception of. In this video for example, I have never heard of SMBs or broadcast joins. This gives me an oppurtunity to learn these things, even hearing them be mentioned from someone as widely experienced as you. You need not necessarily have to even go into detail, but these short form videos act as beacons of knowledge that I can throw myself into learning about. Thanks a lot, and keep these coming Zach!

In the future a wrist watch will have a little blinking light that will have 60 million gigabytes of data in it

Boyfriend simulator: you sit with your bf and he starts talking about this nerdy stuff you have no idea about but need to keep listening because you love him

Thanks Zach, hopefully one day I will understand what all of that means

I love that you kept it short and to the point.

Holy crap. I’m currently learning about data science, the various roles, etc. -with the hope of one day switching careers. But the current state of learning is all about the languages and software used etc, not about the infrastructure and what to do with massive datasets. So this just 🤯

Thank you Zach for taking the time to give us the hard truth and hands down your experience. It helps a lot of enthuastic students/people to know how we can in some way support or help others in the subjects we like. I don't imagine myself processing 2000TBs per day, but it helps give a bigger picture. Once again, appreciate the short video and thank you for sharing

Just started following you. Really appreciate you for sharing your knowledge with the community.

Great content, an honour to be able to listen to someone who has handled that volume of data.

Informative and straight to the point, great stuff as usual

instant subscribe - really appreciate the concise explanation and clear examples

Excellent video, thanks Zach!

Thanks for the info Zach. Could you please make an elaboriative video on SMB join.

I am a regional IT installer who runs Cat6 Ethernet pipelines for managing 1gb loads on HP laptops, this video is really awesome and breaks down your workflow and mindset in a complicated field really efficiently. I would love to get more short videos about the industry like this.

Hey Zach, your content is consistently amazing! As a newcomer to the field, I'm considering diving into data engineering. What roadmap would you recommend, and are there any certifications that could enhance my journey? I already have a solid grasp of Python and SQL in data analysis.

I just found ur stuff but thanks for the content mang keep it up 🙏

I've never heard of these terms, thank you sharing your real case scenarios(The FB notification example)

Half of what you said I had no idea what you were taking about but I was very engaged and now I’m gonna look all this stuff up for centering my div!

Very important concept in such short time.. thank u so very much ❤

Thanks, looking forward to more such content

If you come across a scenario to join 2 large datasets. You could do an iterative broadcast join. Basically you are going the break one of the df into multiple dfs and join the dataframe in a loop till all the multiple dfs are joined.

In the 37 years I’ve been working in data, I’ve never heard anyone call it Peter 😂. PETA

Insightful as always.💯

Please more data stuff!!! I hardly understood what you said, but it’s sounds interesting

The amount of knowledge you shared here is astonishing

Thanks Zach , but I have a question broadcast join is used when we have a small dimensions joined with big table this is your case? Or are you used hash join with two large table?

Dude has beef with Bezos😂

you can make a bios optimized for throughput and without interrupta , to speeden 67x and more

Honestly, 2000 TB per day isn't the problem. The problem is the cost and how much of the data is burst. If it is not burst it is pretty much always cheaper to do it in-house with your own hardware than to pay and rent the cloud to do it.

I would be interested in the architecture and content delivery for pre and post cdn from a network design perspective. Are there any examples or presentations regarding networking at netflix?

Great points to remember! There are a lot more underlying abstraction layers you can add at these different points to further optimize the second network hop. Caching is a simple one. Can you implement an efficient snapshot system with delta encoding of entities and compress the message? Would be a cool video for you to implement!

Thanks for sharing, now I can finally put some good numbers on my resume 🎉

Ah yes, data structures and sorting… but with the “can you even scale bro” tick enabled.

2 pita bites a day, the same as me when I’m on a diet.😊

Please keep up the great content!

Thanks for the video

My problem is how do people even find out about the careers that they go into?

Did Facebook use Databricks or did they have HPC Clusters for you to run Spark on?

Damn I just wanted to shuffle like there’s no tomorrow and then I found this video.

Imma wait for Primeagen to confirm this as well when he reacts to this video inevitably 😁

Never thought broadcast join is a Netflix saviour

I love how you acronym Sorted Bucket Merge as SMB. Think you may have had Super Mario Bros on the mind 😂

Sir this is a Wendy's.

Interesting! I would have thought something like sharding (or partitioning and clustering) so data processing and access can scale horizontally.

Thanks zech for the video

I love that I’m only a software engineer but I can understand all of this

Very nice. Short and sweet.

I’m trying to get into data analytics and most of this we t over my head but this still sounds lit 🔥

I'd like to learn more about these pitabytes. What are they? What do they taste like?

Insanely valuable content

optimizing selling personal data to minimize cost is something i never thought about

Thank you Tony Hawk, very cool!

When I was hired to do data engineering, it was always data that could fit on a single hard drive and it was boring af. I hated it. This sounds way more challenging and interesting.

Hi, what about replacing torrents with IPFS? That's data pipelining, right ?

Managing retention, storage and flow is always important. Im sitting on a toilet as im writing this.

Pretty interesting, even though I had no idea about most of what he was talking about.

That's cool bro. Will it fix the Netflix app where it shows the title of one show but the preview and description of another?

My medical science clients called, they need an 800tb imaging data set parsed by end of day (thank you kubernetes)

Are those joins available in MySQl or specific to dbms at meta you worked?

Very informative, I wanna ask you, which certification can help me as a fresh graduate, is AWS data engineer Certification worth it or not? And thank's a lot Zach

Whenever I hold on to more than 60 petabytes I just call the assistant to the regional manager and he runs a fix from his mainframe.

Hey are you familiar with cosmosDB from azure? Its a db like mongo but claims to be able to scale infinitely... What are your thoughts on that?

Bro can figure out how to send my entire homework folder in 1/500th of a second but can’t flip the camera sideways

Oh yeah that’s really great and insightful, now what’s a join?

I just felt like drinking from the fountain of knowledge and instantly drowning. Definitily haven't had to deal with these kind of volumes yet...

Now I just need a billion dollar company to have these kinda problems. My question would be, why you have table that big? Can't you distribute or cluster your data? I'm thinking like 10000 users per server. Only stuff around those 10k users gets stored. No magic needed to query stuff.

: multiple streams across entire ddrs directly accessible

gotta love a good pita byte

my data pipeline usually processes one pitabyte every other day and one shawarmabyte every week week

Very useful and interesting, even to a layman

The Venn diagram of people who use TikTok and data scientists is two circles my dude lol

Love the way you tried to make it sound more complicated than it actually is and failed.

I might get 5 users on my site this month so this will come in handy

"FNA developer" I'm sorry, my brain couldn't let go of it

The internet is not something you just dump something on, it's not a big truck. It's a series of tubes.

I love technology and I know more than your average user, yet I have no IT qualifications and I am light years away from this knowledge, but for some reason, I love watching these videos as if I was ever going to use the information 😂

Quality content!

Interviewer: name 5 data types Me:

I still bite my gigas when my man hustling meta in peta

This guy earned his half a million salary. I tried to do this myself and failed

I use just a database with just value as field (long string) and nothing else

Bro is the PewDiePie of data Engineering

People don’t know the data they collect is very volatile, unless you are paying for it.

What engine were you using to do these massive joins? Spark?

I don't quite understand why Netflix needs data pipelines.

I love pita bites as much as the next guy, but I don't think I can take more than 35 before I'm full

Yes but why does he look like a French model

ok, so how to do that ...can you make a screencast and show us how to do it!

Me watching this not knowing anything hes talking about makes me feel like starting a big tech company 😀

How do you deal with log data

Short and informative

That's around 160 Gbit/s. Enough for 30K 1080p streams or 10K in 4K.

I built data pipelines at Netflix that ran 2000000000 MBs per day

I suddenly feel like pita bread...

1. Are you a data engineer? 2. What tech is this? AWS, Snowflake?

You shouldn't write "s" in Terabyte per hour, just TB/hr "TBs/hr" looks like "Terabyte*second / hour" 😅

Hey absolutely curious about the content your are doing. In my company we are working dbt and snowflake. I can't find a possibility to work with broadcast joins there. do you see a possibility to replicate this process?

My tech lead keeps talking about bucketing as our integration solution tends to get overloaded sometimes. This kinda puts things into perspective. Definitely dont need most of what he’s talking about but just to know the terms and how to implement them