What is Database Sharding?

Рет қаралды 154,166

Be A Better Dev

Күн бұрын

Пікірлер: 212

@RandomShowerThoughts Жыл бұрын

honestly might be the most complete and thorough explanation of sharding.

@BeABetterDev Жыл бұрын

Thanks so much for your kind words!

@hamadaparis3556 2 жыл бұрын

You've simplified your explanation like google engineers do when they give lectures, I'm sorry if that sounds strange but I've realized that the people who simplify complex things they really know what they are doing awesome man Cheers.

@BeABetterDev 2 жыл бұрын

Thank you so much for the kind words!

@mohamedshehata3979 Жыл бұрын

Nice words

@Aidanhyland 3 жыл бұрын

I am burning through all your videos. You are making me a better SAAS Test Engineer! Keep up this great work!

@dannydatt 3 жыл бұрын

Network guy trying to get an understanding in a different field. That's an outstanding walk-through and very much appreciated. Thank you for your work and quality presentation.

@BeABetterDev 3 жыл бұрын

Glad it was helpful!

@abhishekghosh5550 2 жыл бұрын

This is seriously such a great video man. I spent the entire Sunday understanding Sharding. Not that I didn't get started with the concept, however, this video just made everything clear at the end of the day. Thank You.

@v.m.5850 Жыл бұрын

Watched countless videos and barely understood the concept. Your video on the other hand explained everything along with pros and cons super simply. Thanks a ton.

@rjjlucy 3 жыл бұрын

In most ~20min videos, I get tired soon and close them after 5min. I can’t believe your video is so good that I totally forgot time and finish watching all of it

@BeABetterDev 3 жыл бұрын

Thank you so much Jingyi! Its these kinds of comments that keep me motivated to make more content :) Stay safe Daniel

@MohammedMubashshir-q8v 17 күн бұрын

Awesome explanation of sharding, one of the best videos out there. Thanks brother!

@cd92606 3 жыл бұрын

Great video, especially your description about the non-uniformity problem.

@BeABetterDev 3 жыл бұрын

Thanks Rotary Dialer! Yea the non-uniformity issue is one I've been personally bitten by in the past. Glad you enjoyed the video!

@yfzhangphonn Жыл бұрын

Best lesson about database scalability I found, so easy to understand.

@bharat_arora 3 жыл бұрын

Finally found some decent content over this topic. I already had an idea on this topic just wanted to revise it. Thanks a lot for making the insightful videos.

@filesopen6188 2 жыл бұрын

this video entails very good explanation and this also entails complex understanding.

@BeABetterDev 2 жыл бұрын

Thank you!

@codespace747 4 ай бұрын

Best video ever made on sharding

@ase713 4 ай бұрын

Dude, this was outstanding! Super helpful and covered everything I needed to know!

@poloska9471 2 жыл бұрын

Dude you make some really awesome content. Please please keep making videos! I love the clarity of your speech, voice, and presentation. I understand and can follow along in your videos a lot better than more other channels. Earned my subscription and likes! Keep killing it homie!

@BeABetterDev 2 жыл бұрын

Thank you so much for your kind words and welcome to the channel!

@Bhaskarlnm 2 жыл бұрын

Daniel, no words.. looking at your playlists content and videos …amazing. Great great effort to help people. Kudos to you 👏👏👌👌👌

@BeABetterDev 2 жыл бұрын

You're very welcome!

@harishbendale6818 Жыл бұрын

Very clear, and simple explanation.

@BeABetterDev Жыл бұрын

Glad it was helpful!

@devdewboy Жыл бұрын

Thanks for the straight forward easy to grasp concept of sharding. Give this to someone else and we would have gotten a bunch of technical wordy mumbo-jumbo.

@Alexan6548 Жыл бұрын

Very clear. One of the best tutorial I have ever seen

@saiaussie 5 ай бұрын

Hey dude, you're a star! Very clear and upto the point! I cant thank you enough.

@bahot9057 Ай бұрын

Very well explained, great video!

@JayPatel12928 2 жыл бұрын

Watched some of your random videos on sys design, and now im hooked. Great content!

@BeABetterDev 2 жыл бұрын

Thanks so much J! Glad you enjoyed!

@amrikmalhans 2 жыл бұрын

@@BeABetterDev yes

@IQUE928 2 ай бұрын

incredible explanation, thank you!

@mathisinav4267 2 жыл бұрын

Hands down! the best explanation I've seen on database sharding, excellent!

@BeABetterDev 2 жыл бұрын

You're so welcome. Glad you enjoyed.

@JamesQQuick 2 жыл бұрын

This was awesome. Thanks!

@eugeniosp3 2 жыл бұрын

Bro I'll watch anything you make. If you made a video teaching me how to watch paint dry I'd take notes. Keep up the damn good work my mans.

@user-zr4jl4ms3v 2 жыл бұрын

best explanation of sharding i've heard!

@BeABetterDev 2 жыл бұрын

Thank you very much!

@wlcheng 2 жыл бұрын

Great video! Such a clear explanation of how database sharding works.

@ChauDuong1982 3 жыл бұрын

Thanks for the videos. Great explaination.

@mivel9763 2 жыл бұрын

Had a hard time grasping on what database sharding actually meant but your video really helped me understand it, thanks! :)

@BeABetterDev 2 жыл бұрын

You're very welcome!

@Lordnoashi 2 жыл бұрын

Amazing explanation, loved it. Thank you, it will help for the future interviews I have.

@BeABetterDev 2 жыл бұрын

Glad I could help!

@ihateorangecat Жыл бұрын

Best of the best explantion I kinda start to understand the Sharding in just video.Thank You and I subbed!

@patrick1778 2 жыл бұрын

you are so good at explaining concepts

@Anton_Rozhanskii 3 жыл бұрын

Great explanation, Daniel. Thank you

@BeABetterDev 3 жыл бұрын

You're very welcome Anton!

@lucasarbex926 2 жыл бұрын

Great content man!! It helped me a lot!! Keep up with the good work!

@BeABetterDev 2 жыл бұрын

Thank you!

@eternalnight9453 3 жыл бұрын

New here. Loved your talk! Your presentation and teaching is elegant and simple. Really appreciate it, thank you!

@BeABetterDev 3 жыл бұрын

You are so welcome!

@OscarMartinez-nt6zn 4 ай бұрын

Great explanation!

@arikedada Жыл бұрын

great video, I understand what idempotency operations entails, thank you

@quang.luu.179 Жыл бұрын

Good stuff man. I love the clarity you bring to a subject. Subscribed.

@donaldkennedy7993 Жыл бұрын

superb explanation of DB scaling & sharding & W/R databases for a non DB person ;)

@tamaraamanda2483 2 жыл бұрын

Prepping for Amazon TPM interview and this is so helpful!

@BeABetterDev 2 жыл бұрын

Thanks Tamara and good luck on your interview! Make sure you focus on those leadership principles !

@chandnisaini9176 2 жыл бұрын

Well explained!!

@BeABetterDev 2 жыл бұрын

Thank you!

@saifmohamed1776 3 жыл бұрын

which better to start with for database basics: - introduction to database systems c.j date . - database internals. // if there are any better or recommended books or materials pls mention. * Great explanation.

@BeABetterDev 3 жыл бұрын

Hi Saif, This is a tough question to answer. I would step back for a moment to ask why are you trying to learn about databases? I think the answer will guide how/what to tackle first. For example, if you're just planning on using dbs, the database internals may be a bit overkill (but good to know overall). Could you tell me more about why you're learning db's and maybe I can guide you more? Thanks, Daniel

@saifmohamed1776 3 жыл бұрын

@@BeABetterDev to be aware of the basics in general like concepts physical logical at first And in backend specific. I'm very grateful for your concern

@BeABetterDev 3 жыл бұрын

Hi Saif, I briefly looked at the two resources you mentioned, I think a better choice is to read Database Internals. I feel that it is much more modern and covers some of the important aspects of database challenges today such as distributed systems and availability. The other book is quite dated and although I'm sure would be beneficial, I think things have changed so rapidly recently that I'm concerned the content will be a bit stale. One thing to note is to not get too bogged down with the details. To be a great developer with database understanding you don't always need to understand the low level details. Knowing how things work at a high level with the ability to dive deep when you need to is much more valuable. Hope this insight helps and I wish you best of luck on your studies. Daniel

@saifmohamed1776 3 жыл бұрын

@@BeABetterDev thank you

@rajt1998 2 жыл бұрын

Very well explained. Thank you

@BeABetterDev 2 жыл бұрын

You're very welcome!

@sharonleibel 2 жыл бұрын

Great explanations! Thanks, Keep it coming!

@BeABetterDev 2 жыл бұрын

Thanks Sharon!

@santoshlml 3 жыл бұрын

Well explained. Thank you!!

@BeABetterDev 3 жыл бұрын

You're very welcome Santosh! Glad you enjoyed.

@taniaasim 2 жыл бұрын

This is great and super clear. Thank you!

@BeABetterDev 2 жыл бұрын

You're very welcome

@random-characters4162 Жыл бұрын

God bless you, sir ✌️

@RajuGupta-st1hj 2 жыл бұрын

Thank you so much for the post. Good work. Keep it up.

@BeABetterDev 2 жыл бұрын

You're very welcome Raju!

@estebanquintana156 2 жыл бұрын

Great explanation. Thank you

@BeABetterDev 2 жыл бұрын

You are welcome!

@shubhamchandra9258 2 жыл бұрын

Awesome video. Very clearly explained.

@BeABetterDev 2 жыл бұрын

Thanks!

@jackforcecity 2 жыл бұрын

Great job. Very well explained!!!

@BeABetterDev 2 жыл бұрын

Thanks so much Jackson! Glad you enjoyed :)

@rschmidtzalles 3 жыл бұрын

clear and concise. subscribed

@BeABetterDev 3 жыл бұрын

Thanks Libert and welcome!

@milequinze 2 жыл бұрын

Awesome! Thanks a lot!

@BeABetterDev 2 жыл бұрын

You're very welcome

@duzgunkenan 3 ай бұрын

Great content, thank you

@nodrift9503 Жыл бұрын

Perfect explanation. Thank you

@andrewkicha1628 Жыл бұрын

Great job on this one, I came here to know more about sharding, but I learned lots of useful information before you even dived into the topic ;)

@BeABetterDev Жыл бұрын

Glad it was helpful!

@panggrayta 3 жыл бұрын

woww...!! great videos, great presentation, great explanation. thank you, keep sharing..

@markryan430 Жыл бұрын

OMG, I just Sharded myself!

@itiscinnamoncafe 11 ай бұрын

Love longer videos ❤

@hpandeymail 2 жыл бұрын

Very well formed content .. thanks 🙏

@BeABetterDev 2 жыл бұрын

You're very welcome!

@royal4210 2 жыл бұрын

Thank you programmer charlie

@dorbenmoyal7639 3 жыл бұрын

Great explanation..! Thanks 🙏🏼

@BeABetterDev 3 жыл бұрын

You're very welcome Dor!

@willemplug3366 Жыл бұрын

Super clear. Thank you!

@drew4980 3 жыл бұрын

Are there any database tools that make this easier? Couldn't someone write some software to create a wrapper around a sharded DBMS that could handle the routing and re-sharding with a given hashing key?

@SofiaGoyal 3 жыл бұрын

Really good work man... such a detailed video...

@BeABetterDev 3 жыл бұрын

Thanks Sofia! Glad you enjoyed :)

@jeevaengg Жыл бұрын

good explanation!!

@samlinsell900 2 жыл бұрын

Vids are awesome, really enjoy them. Interesting that you didn't touch on the lack of thought to database design, indexing and maintenance etc as a way to improve performance. Interested to know why? Especially given the cost of scaling in serverless environments.

@user-ws5xs1nv4p Жыл бұрын

Thanks a lof for this video!

@3042640426 Жыл бұрын

Thank you so much

@AnilKumar-lb3qf 2 жыл бұрын

Excellent presentation, very good explanation 👍👍

@rayprusia4753 3 жыл бұрын

Your videos are awesome! Thanks

@BeABetterDev 3 жыл бұрын

Thanks ray!

@subhasishhalder4817 2 жыл бұрын

How come I didn't find your channel before?

@francoisbourdages2906 Жыл бұрын

très bien expliqué, merci

@trantrongty8065 3 жыл бұрын

Thank you that really helpful great video

@BeABetterDev 2 жыл бұрын

You're very welcome Tran!

@sandrojorgeoliveira175 Жыл бұрын

Thank you, Dude!!!

@rishiraj2548 Жыл бұрын

Thanks

@user-tu1kf6xd8d 3 жыл бұрын

great explanation thank u so much

@BeABetterDev 3 жыл бұрын

You are welcome!

@simonemariottini1011 3 жыл бұрын

Really useful content! Keep it up!

@BeABetterDev 3 жыл бұрын

Thanks so much Simone!

@Tiparium_NMF 2 ай бұрын

I love this breakdown, but it does somewhat leave me wondering when Sharding would be a good vs a bad idea. The cons seem pretty hefting in comparison to the pros. It would have been nice to run through a few specific different use cases and when one strategy would be better than another.

@kgcpk 2 жыл бұрын

Superb explanation 😍

@SuperWhatusername Ай бұрын

Great content

@drummerboi4eva Жыл бұрын

excellent video

@geekaffairs6475 2 жыл бұрын

very well explained

@BeABetterDev 2 жыл бұрын

Thanks so much!

@dushyantchaudhry4654 Жыл бұрын

questions: 1. Database is a slightly misleading term.. when we say database don't we really mean the software (RDBMS / NoSQL) that logically organises the data stored in storage SSDs? 2. If yes are we not splitting the responsibility of the software? i,e. The data still is in the SSD library right? Just the database management software is loaded in different servers and each DBMS server given responsibility for only some of the queries.

@ibrahimkoz1983 2 жыл бұрын

You're great.

@BeABetterDev 2 жыл бұрын

Thank you!!!

@atacoonthis 3 жыл бұрын

This dude is the Cr1tikal of databases

@BeABetterDev 3 жыл бұрын

Haha thank you Daniel!

@gunnerandersen4634 2 жыл бұрын

Hi there, a nerdy engineer here, I think you could bind some kind of hash or unique ID to your queries, and then track this as a state, then you can verify that a replica has gone through that particular state to make sure the values you are using are at a valid state for the new given query. Not saying this is any good at all, don't get me wrong, as a matter of fact, I think performance-wise might be actually very bad, but it kind of solves the event problem somehow.

@benhunt2023 2 жыл бұрын

This feels like the khan academy of coding. Well done.

@r-rtz 11 ай бұрын

A more interesting concept though is how you generate these unique id's that are used in the sharding / partitioning and ensure uniqueness

@kishoretvk 2 жыл бұрын

Great vidoe, keep up the good work

@vikasravindran1312 2 жыл бұрын

Excellent !

@BeABetterDev 2 жыл бұрын

Thank you so much Vikas!

@poketopa1234 3 ай бұрын

What I always miss in these videos is, doesn’t introducing a routing layer just kick the can down the road? Now you have all traffic going to a singular routing node, which is not scalable and can fail. What happens when you need to scale the routing node?

@shivakumarranjithn5584 3 жыл бұрын

Great content!

@BeABetterDev 3 жыл бұрын

Thanks so much Shivakumar!

@skmahaboobbasha6059 2 жыл бұрын

Great vedio please make vedio on opsmanager installation on production environment

@channuangadi7504 8 ай бұрын

and there is another complex thing is the id generation (here Customer ID) when we shard we have to make sure duplicate ID should not be generate, can we have video on ID generation in distributed computing

@gulsimsek7988 3 жыл бұрын

thank you!

@vanchark 10 ай бұрын

This is one of the best explanations that I've seen on this topic, but it still isn't really clear to me how the problem of eventual consistency that you see with the single master-slave paradigm is fixed with sharding. It seems like the problem is still there, since even with shards this master-slave paradigm still exists, just on a smaller scale now

@cyclomiha 5 ай бұрын

Hmm..how about PITR? For analytics you could have replica with multi-master approach to each shard, right?

@peterroger249 Жыл бұрын

Much thank you for your great KZfaq help. I am new to Excel and Chatbot. How can I migrate the Excel database, export it from Microsoft Azure WebApp, and import it into AWS Chabot? Keep having errors missing QID and others on the AWS Chabot console. Please help show me the fastest way to convert the Excel and make it compatible with AWS Chatbot?