How Discord Stores TRILLIONS of Messages

Рет қаралды 656,761

Күн бұрын

Have you ever wondered how Discord is able to store trillions of user messages without breaking a sweat? In this video, we'll take a deep dive into the technologies and strategies that Discord's software engineers use to manage such a massive amount of data.
SOURCE: / discord
Discord is one of the most popular communication apps in the world, with over 250 million users and counting. With such a large user base, Discord's servers receive an enormous amount of messages every day. To store all these messages, Discord employs a distributed database system that runs on hundreds of servers worldwide.
This system uses a variety of modern database technologies, including Cassandra, Redis, Elasticsearch and most recently: ScyllaDB to ensure that messages are stored reliably and efficiently. These databases are designed to handle huge amounts of data and provide fast read and write capabilities, making it possible for Discord to scale to millions of users without compromising performance.
So if you've ever wondered how your messages are stored on Discord, or if you're just curious about how modern database technology works, this video is a must-watch. Join us as we explore the world of distributed databases and learn how Discord's software engineers keep trillions of messages organized and accessible for millions of users around the world.
👻 MY NEW STARTUP: www.caspr.ai/
MY NEWSLETTER 💌
thebetter.dev
------
CONNECT WITH ME ON SOCIAL
📸 Instagram:
/ lewismenelaws
🎚TikTok:
/ lewismenelaws
🐣 Twitter:
/ lewismenelaws
--
My gear 💻
liinks.co/lewismenelaws
-----
TIMESTAMPS ⏰
0:00 Intro
0:38 The BackStory
1:46 Cassandra Troubles
4:09 Change Architectures
6:28 Data Services
8:57 The BIG Migration
10:35 Where It's at now
11:40 What we can learn from this
12:46 Thanks for watching and subscribe :)

Пікірлер: 585

@CodingWithLewis Жыл бұрын

Thanks for watching this video :) What company would you like me to cover next?

@infinitelancer4612 Жыл бұрын

@ExplodeCode Жыл бұрын

Do How ExplodeCode is cool. naa do snapchat or instagram or something based with pictures

@adill6295 Жыл бұрын

Rust roadmap

@akvamsikrishna5535 Жыл бұрын

How telegram makes money and how it stores as much as data like discord and much more (you too will get more questions once you start to think how telegram is able to do it 😅)

@akvamsikrishna5535 Жыл бұрын

And 😅 I want you and forrest knight to have a video together 🥺

@puspamadak Жыл бұрын

The engineering was truly awesome. Have to salute the engineers as well as SyllaDB team for such a great product.

@demin-e Жыл бұрын

No © symbol next to “Rust”? Lawsuit is already being filled

@jamoncitovideos Жыл бұрын

This types of videos are great. It's not common to face scenarios that have this massive scale and it's great to get to know how they have been handled before by other engineers

@ryojikn Жыл бұрын

This is amazing, please keep going. Such an amazing source of knowledge and inspiration in a perfect method for better engineers

@5095smf Жыл бұрын

Nice! I love databases and learning more about them, great video.

@coolxify Жыл бұрын

😂 same

@staticobjx7232 Жыл бұрын

Hi Lewis, I discovered your channel thru YT Shorts, but I like this format a little better, not only it dives into architectural problems and solutions, in my opinion it is not rushed. Also, awesome graphics! Thanks a lot and more power to your channel.

@nerduser1000 Жыл бұрын

The type of content I love to watch thank you lewis! Had this question for Reddit how reddit saved and would love to know how discord works

@wonkywonky6307 Жыл бұрын

This is fantastic! Congratulations to all those amazing engineers and congratulations to you for making this great video. Thank you for bringing this content to us.

@vaggelis_best Жыл бұрын

11:24 Poor Descord engineers😢 They are stressing even HARD

@Raul-pg1pf Жыл бұрын

lmao

@sorteslyngel2k Жыл бұрын

How the hell is discord even making money. That infrastructure sounds pretty expensive...

@agentnull5242 Жыл бұрын

Nitro 😭

@agentnull5242 Жыл бұрын

Bloody expensive too

@lucachiapello6454 Жыл бұрын

Data

@tonik2558 Жыл бұрын

Venture capital is one hell of a drug

@MtTheToto Жыл бұрын

They are not making money. Welcome to capitalism

@AdoSean Жыл бұрын

Really fascinating video and well made. Thank you!

@Luxcium Жыл бұрын

I am so happy 😁 to see how it goes with your channel and how much your topic is related to more advanced topics

@walterrodriguez9550 Жыл бұрын

great breakdown and description, thank you!

@nemzyxt Жыл бұрын

Whoa! This is so enlightening, thanks Lewis! The engineers are doing really great work bts

@NaviYT Жыл бұрын

I was today years old when I learned that Lewis makes long form content.

@TheMassgames Жыл бұрын

This types of videos are great, and your animations are really on point

@My.MusicChannel Жыл бұрын

This video was perfect omg!! Liked & subscribed, keep it up ❤️❤️

@BrendAn_RSA Жыл бұрын

This is my first video on your channel. Loving it. Awesome video :)

@TNTISGOOD 5 ай бұрын

agreed, my first video too! :D

@joshburger2217 Жыл бұрын

as a dev, just thinking about a migration of that scale is anxiety inducing

@cocoiyeah Жыл бұрын

Kudos to the ScyllaDB team! I've been a fan of their product since 2019, and in my opinion, it's better than Apache Cassandra. However, their open-source license limits usage to only five nodes.

@Jecsham Жыл бұрын

the ydeserve being paid for that product

@dominikmankowski1636 Жыл бұрын

Not really. The 5 nodes limit applies only to ScyllaDB Manger, not the OSS database.

@pietrodc0 Жыл бұрын

72 nodes per 140mln users = ~1.9 users per node. 5 nodes limit = ~7.72mln users = you must have money for DB licenses 😅

@OT-tn7ci 11 ай бұрын

@pietrodcfab but that's not how it scales

@ryanzspaulding 7 ай бұрын

I like your funny words magic man. Definitely didn’t understand about 95% of this video but it was really fascinating and well presented

@justingolden21 Жыл бұрын

Love the video and also the short takeaways at the end

@xenofenus 11 ай бұрын

This was an incredibly well made video!

@smixqse Жыл бұрын

i love the way the subtitles interpreted scylladb: s db, mes db, solid b, sillydb, facilitydb, silladb, sil a db

@thewhitefalcon8539 Жыл бұрын

still a DB vs no longer a DB

@bvorg Жыл бұрын

Everything is perfect. Thanks much Lewis!

@peckychicken Жыл бұрын

Wow discord are really committed to keeping their app up to date

@Fingiorstyle Жыл бұрын

Thank you for this video! Please create more

@weiSane Жыл бұрын

I'm waiting for the day discord switches to vector databases when they have to store zetabytes of messages 😂

@nuvotion-live Жыл бұрын

I can see it happening. I’m sure the singularity will shake up technology a lot, and it does seem like vector databases will continue to become more popular

@ThatBubbledGuy Жыл бұрын

Your channel needs to be more popular. This content is the only thing that motivates me to write some code after running into an error.

@costelinha1867 Жыл бұрын

Bold of you to mention the Rust language and use it's logo in this video. Considering what the Rust Foundation has been up to recently.... AND YOU SHOW A PICTURE OF THE LOGO IN A DIFFERENT COLOR... oh my god, I hope the foundation doesn't see this.

@CodingWithLewis Жыл бұрын

🤣🤣🤣

@bobtheoutbuilder8552 Жыл бұрын

I can imagine reading discord messages from 47 years ago.

@denitechYT Жыл бұрын

Aha so now we know how! Great video man!

@def1nt Жыл бұрын

Very cool format and interesting stories! I wish Discord to find frontend devs as good as their backend team

@AnushR27 Жыл бұрын

Great video Lewis 😃. Got to know so much information about Discord databases. Thankyou!

@Skeffles 6 ай бұрын

Great video. Fascinating to see how they solved this!

@felipheleal Жыл бұрын

Excellent video 👏🏼👍🏼

@shloktadilkar5536 Жыл бұрын

Very nice video Lewis

@nickhtt 6 ай бұрын

What a nice video. Thank you very much :)

@Keys_02 Жыл бұрын

Great content, please I want to learn about the internet from a developer's perspective what book are you going to recommend for me?

@nathan0401_ Жыл бұрын

This was very fascinating to watch

@roshangamage7749 Жыл бұрын

Great video ❤

@somnathroy102 Жыл бұрын

Thank god yt recommended this. I learned a lot.

@YosephTeki Жыл бұрын

wow very very insightfull !!! thanks for making this video

@JoeyClover Жыл бұрын

The concern you described at the end is a well known anti-pattern called the Golden Hammer. Don't get too comfortable with the tools you know and use.

@frogery 9 ай бұрын

love the animations on these videos

@AhmedMi1ad Жыл бұрын

Wow I really enjoyed the video, so insightful.

@joker-wr8pt Жыл бұрын

Sir you're my inspiration these types are useful please bring more like this like how big system works so that we can have real-time understanding

@bajagainsatish4828 Жыл бұрын

Awesome dude!!!

@MultiSciGeek 9 ай бұрын

This is better than ASMR. Fell asleep in 8 minutes!

@ninjanape Жыл бұрын

Great video. Subbed.

@avonzo 11 ай бұрын

Great video. Great content. Thank you.

@IgnitedIce81 Жыл бұрын

Never realized how incredible the engineering is on Discord

@IRazerIGhost Жыл бұрын

Great video!

@CodingWithLewis Жыл бұрын

Thanks for the visit

@ojasmittal8381 Жыл бұрын

This guy basically turned Discord's blog into an interesting video 10/10.

@po6577 Жыл бұрын

this story are amazing, great motivation to improve myself!

@MistaWu 10 ай бұрын

That was cool… awesome breakdown…🔥🔥

@sebobebo10 Жыл бұрын

I read the title and immediately thought, “I wonder how much storage space I can take up by just spamming emojis…”

@Joshuahendrix Жыл бұрын

Really enjoyed this one ☝️

@-hackers_industry Жыл бұрын

Could you do "A roadpath to become a software engineer" some time soon?

@varshard0 Жыл бұрын

So informative and entertaining.

@awsomevideoperson Жыл бұрын

I wonder if the issue with the normal Cassandra is really the GC, or the memory pressure Java currently has due go reference types. I suspect user defined primitives and fibers will really make Java a stronger contender in the high performance space

@Terracronz Жыл бұрын

Cool video, earned a sub

@lisibethklair 10 ай бұрын

every time I think about discord I get a horrendously awkward flashback to my first situationship :/ when he asked “do u have discord” I replied “about you staying or about you leaving?” 💀

@DemiImp Жыл бұрын

You just altered the color of the rust logo. That is in violation of their trademark!!! 😂

@allezvenga7617 Жыл бұрын

Thanks for your sharing

@tanmaypanadi1414 Жыл бұрын

the editing on this channel has exponenetially increased in quality 🤩

@willmil1199 10 ай бұрын

Love this channel

@kumalalalalalalaa7045 Жыл бұрын

Love these videos

@farhan-momin Жыл бұрын

what a video! learning + entertainment

@amogid Жыл бұрын

Lewis: *thinking he says NoSQL bad* Me: *says NoSquirrel*

@pemessh Жыл бұрын

Wow this is an awesome informative video. Please do keep them coming!!

@javadevelopment6551 11 ай бұрын

Great video, and I'm left with one question how to find and use the right tools at the right place and at the right time 😅 That something comes with experience or any beginning can also master that?

@musikSkool 8 ай бұрын

Discord is the perfect combination of Twitter, Facebook, an in-game chat bar, and even Instagram. If you pay them a little you can even treat it like KZfaq or Twitch.

@zuman_one 7 ай бұрын

Please Keep making such system design videos because I don’t like to read long articles 😂

@Vergilenjoyer Жыл бұрын

Subtitles: "Hi, my name is, and" Video: "Hi my name is Lewis"

@mukithasan9684 Жыл бұрын

Today i learned a lot about database management how behind the scenes work

@vncntjms 8 ай бұрын

"what any programmer would want to do when they see something new: changing architectures" I'm guilty.

@simonkimel3339 Жыл бұрын

This shows that Discord developers are really capable, yet the auto update on Linux doesn't work.

@Tudas Жыл бұрын

Because Linux is literally completely irrelevant, atleast on the consumer market.

@oh-noe Жыл бұрын

@@Tudas lol so dumb

@lucass8119 Жыл бұрын

@@Tudas You would think Discord, a company that relies on 100% of their own servers running Linux and is a consumer of open-source products like Apache Cassandra, would put just a tad more care into the linux version.

@Nitidus Жыл бұрын

@@TudasI mean, there's the Steam Deck now. It definitely has become a LOT less irrelevant compared to a couple of years ago.

@umaradam3788 Жыл бұрын

@@Tudaswhat are you talking about. The entire consumer market is reliant on Linux. Facts.

@MyNameIsSalo 6 ай бұрын

have never put much thought into discord but they really are at the peak of social media, almost 0 downtime ever and always provides a consistent streamlined experience. I can imagine that they will end up doing many many more database migrations in the future as discord dominates the social media space. Wonder if there will ever be a permanent solution of a database which is truely infinitely expandable.

@youtubear2217 Жыл бұрын

"2017 was 6 years ago" nooooooo😭

@bideshbanerjee5506 Жыл бұрын

I may be wrong but I heard from someone in a livestream that data service layer can be implemented in golang and the latency difference would still be unnoticeable

@andixpiwi8325 Жыл бұрын

discord working their ass off just to allow me and my friends to write the dodgiest messages ever (we're on a government watchlist)

@adityatomar-5308 Жыл бұрын

😂😂

@rkubapl 7 ай бұрын

great video

@aze4308 Жыл бұрын

Great video.

@orlovchik Жыл бұрын

thanks for the video!!! the question was in my mind this morning and, what a great recommendation, you gave me a full answer! but I have one more question why discord mobile app is so laggy and slow, while its web and desktop versions works fine

@TomWhi Жыл бұрын

Interesting video. I feel sorry for that underused blackboard though… it looked like it was going to be written on about 5 times with not a single mark ever made on it!

@Benoit.mp4 Жыл бұрын

Very interessing !

@skiper7795 Жыл бұрын

I love discord. Best comunicator out there

@god_of_everything9644 10 ай бұрын

Got urself a sub sir

@SusGus-rf8gm Жыл бұрын

A few years ago, I don't remember why exactly but I had issues my phone plan and I was stuck with worse than dial up speeds for a while. Im talking low kilobytes per second, maybe even bytes. Discord was literally the only "online" app that still worked pretty much normally (besides embeds). That thing is optimized to death

@harrywang4769 Жыл бұрын

yeah that's not it. the size of the data being sent and received has no correlation between how optimized the app is. your internet connection works for discord because the amount of data sent by discord is very small. it has nothing to do with app optimization.

@TheDingsBoms Жыл бұрын

Inspiring!

@404statuscode Жыл бұрын

Make a video on how Google sheets can handle Sheets that are thousands of rows long without even the user noticing it. For example How it can load the data faster(even with my crappy internet). I know that they will be using compressions and things. But a video or a short would be nice explaining it.

@martinmc0950 Жыл бұрын

It’s just text. Your average KZfaq video is harder to load

@Tudas Жыл бұрын

@@martinmc0950 It's not just text lmao, did you ever see Sheets that are fking huge? But Excel does that even better, NASA literally uses Excel for huge ass calculations.

@martinmc0950 Жыл бұрын

@@Tudas just because the sheets are big doesn’t mean it’s not text??? Although I see where you’re coming from with the whole calculation thing, using a computer for math is always going to be effective

@infinitelancer4612 Жыл бұрын

That's awesome

@CodingWithLewis Жыл бұрын

More than awesome!

@itsjustsushi2023 Жыл бұрын

This could've been the perfect video to advertise your dc

@thewhitefalcon8539 Жыл бұрын

He's got his own data center? Nice

@blackbriarmead1966 Жыл бұрын

I love how you made the letters of coalescence alternate purple and yellow... like Moira's ultimate in Overwatch. This was intentional, right?

@CodingWithLewis Жыл бұрын

Hmmmmm 🤔

@darpitrangari8207 Жыл бұрын

Can you go into a little more details as to why they were facing difficulties?

@vanilla-plus Жыл бұрын

Interesting colour choice on the Coalesce... What an interesting hypothesis.

@Loading_Code Жыл бұрын

I only watch your videos because your expression are amazing

@27.Counting Жыл бұрын

I have a questuon ... If they had written the data services first , would they still migrate from cassandra ? Considering that daya services was their bigger improvement ?