Designing INSTAGRAM: System Design of News Feed

Рет қаралды 616,483

Күн бұрын

This video is about designing Instagram for a system design interview. We are expected to design the server side so that all 4 mentioned features can be incorporated. System Design interviews evaluate a candidate's computer science fundamentals, architecture principles, etc.
We start by building an entity relationship diagram. Here, the columns are defined as per the requirements. We then move to designing a microservice architecture. Each microservice represents a single component of the system.
The newsfeed is built step by step with concepts like load balancing and caching as prerequisites. This completes the system design for Instagram.
Looking to ace your next interview? Try this System Design video course! 🔥
interviewready.io
With video lectures, architecture diagrams, capacity planning, API contracts, and evaluation tests. It's a complete package!
System Design playlist:
• System Design BASICS: ...
Some of the concepts discussed here:
Tinder video: • System Design: TINDER ...
Database reference: www.db-book.com/db7/
Load Balancer: • What is LOAD BALANCING...
Consistent Hashing: • What is CONSISTENT HAS...
Publish-Subscribe: • What is the Publisher ...
Chapters
00:00 Introduction
00:10 Feature Selection
03:15 DB Schema
09:45 User Followers and Following
12:02 System Design
21:32 Celebrity post fanout
#SystemDesign #DesignInstagram #DesignInterview
Become a channel member!
/ @gkcs
You can follow me on:
LinkedIn: / gaurav-sen-56b6a941

Пікірлер: 590

@elliemay1748 4 жыл бұрын

I interviewed for PayPal a few weeks ago, and this was the exact systems design question they asked. They said to me, “Do a system design for Instagram”. I smiled, because I had just watched this video a few days prior, and so I knew exactly how to answer. Thank you for this video, you helped me get a job, for real :)

@gkcs 4 жыл бұрын

Congratulations!

@sonushete4431 3 жыл бұрын

wow awesome Elli May got job in Paypal 😁

@brandonzheng1092 3 жыл бұрын

What's there to be proud of when you've seen the answer to an interview question beforehand?

@ashishsharma-nz5pq 3 жыл бұрын

@@brandonzheng1092 so one should not feel proud anyways, cuz he/she had studied that in books b4... lame perception

@pratik3106 3 жыл бұрын

@@brandonzheng1092 I sense happiness rather than pride, none of that this person said implies proudness, and even if so, why not? I'd be proud.

@Tuanvu-tb1mh 4 жыл бұрын

Hi Sen, for the database design I think you should go from Logical ERD first then derive Physical Tables from there, it is more natural approach

@samanrajaei8129 3 жыл бұрын

Damn this kid is good. Better than most of the "veteran" system architects I've worked with.

@dijoxx 3 жыл бұрын

You must have worked with some really crappy architects if that is indeed the case.

@apurvsawant5703 3 жыл бұрын

All the important concepts are explained very simply and this is what makes this video amazing.

@AtomicAkshay 3 жыл бұрын

Great video man, really appreciate the fact that you've been posting such indetail conceptual content for free.

@theakatsuki2113 2 жыл бұрын

Wow his content is really at next level I love this guy and respect his efforts and the amount of hard work he puts in each and every video

@HarkiratSaluja 4 жыл бұрын

Probably the first video of the system design I have seen. Being a front end developer, I had a fair idea about the things but the way you explained just wow

@ChitranshuVashishth 5 жыл бұрын

I'm not even a CSE student. But watching your videos, Gaurav, actually intrigues a lot and motivates me to actually learn more about programming and design some my own scalable system one day. Thanks Gaurav.

@blasttrash 5 жыл бұрын

Try coding train channel. You can watch most of his videos and have fun. Its like watching a movie and he does real coding. For example watch this video, even if you dont know programming, you will still understand it and its so much fun. Coding the snake game kzfaq.info/get/bejne/d8d3fpCXz5OmcYE.html

@xReisk 4 жыл бұрын

As a backend developer who strugles to do projects due to thinking to much and taking a lot of time to make dumb things... I think I have found one awesome channel for me. Thanks for your videos man!

@gkcs 4 жыл бұрын

😁

@amlanroy5274 2 жыл бұрын

Well, I wasn't even looking for this. Just got a random recommendation and now I'm watching this with full focus at 3AM. Never thought this day would come😂 Awesome video.

@bmw1553 4 жыл бұрын

Gaurav, first of all, thank you for a fantastic, simple and clear explanation. Second of all, I can imagine the work went in to put this video, it must be humongous task in preparing the right content, taping, editing, etc. Great work!

@gkcs 4 жыл бұрын

Thank you!

@ci7alex1 4 жыл бұрын

Explains a lot why there are so many well paid people behind each successful online service, so complex, wow

@aditikhedkar8514 5 жыл бұрын

I always wait for your videos!very good content Thankyou!! :)

@HemantSharma-fw2gx 5 жыл бұрын

Thanks for explaining the practical use of all we study in our syllabus..Your videos are superb!

@gkcs 5 жыл бұрын

Glad to hear that!

@yog2915 4 жыл бұрын

Fantastic work it is because of people like you skills of general masses are also rising

@johnfrades 3 жыл бұрын

Awesome man, im glad to see your channel, subscribed immediately! Very helpful!

@augmentos 3 жыл бұрын

Would love to see a system design of notifications (activity feed) in twitter/IG etc. Aggregate etc.

@devarajt6965 5 жыл бұрын

hi Bro.. Actually the way you explained the stuff is very simple and clear.. Thanks for your time for making such videos..

@gkcs 5 жыл бұрын

Thank you!

@alexkuzmichev9051 4 жыл бұрын

I'm so glad I found your channel. Keep up the good work! Nice videos:)

@gkcs 4 жыл бұрын

Thanks 😁

@nitinjoshi4636 5 жыл бұрын

Awesome content.. what I am looking for always get from your videos. Keep it up.

@antarikshsrivastava2562 4 жыл бұрын

Absolutely loved your explanation Gaurav. Thank you :)

@ankitgoyal8556 4 жыл бұрын

Loved this playlist, thank you brother

@AlphaWatt 2 жыл бұрын

Thank you very much for this. Excellent explanation through such a complicated topic. Really helped me think through a follower service I have been struggling to commit to.

@harkirat1 5 жыл бұрын

Great Explanation. Clear and concise.

@gkcs 5 жыл бұрын

😁

@sankalparora9374 Жыл бұрын

Amazing stuff - not only informative - but interesting! Thanks!

@A.n.a.n.d.k.r. 4 жыл бұрын

I didn't knew the dbms subject was so much exciting....

@dbtechprojects2392 3 жыл бұрын

Thanks for posting this, the part on how to handle the news feed helped me out a lot, originally I could only think of the first method which the administrative tasks are way too high, precomputing the news feed is an option I didn't even think about. thanks :)!

@gkcs 3 жыл бұрын

Glad it helped 😁

@code_report 5 жыл бұрын

Great video!

@gkcs 5 жыл бұрын

When code_report says something, you better believe it 😎

@bianbian621 4 жыл бұрын

great video! the animation part is awesome. I like all your system design videos.

@gkcs 4 жыл бұрын

Thank you 😁

@yuxiongzhu4249 5 жыл бұрын

Hi Gaurav, thanks for this great post. You look so young, how could you be so knowledgeable?

@vikaspizza 3 жыл бұрын

He is actually 45 years old. He designed a system that removes aging signs from his youtube uploads...

@emmanuelevbuomwan2665 4 жыл бұрын

Concise, at the same time; broad and easy to understand.

@gkcs 4 жыл бұрын

Thanks 😁

@shradheytripathi7564 5 жыл бұрын

Loved the explanation :) Nice work.

@gkcs 5 жыл бұрын

Thank you!

@nayankhanna2367 3 жыл бұрын

Been addicted to this channel recently and binge watching it even though I have my exams ongoing xD. Man you are the best. (saying this from my experience of having watched more than 100 "Real" Coding KZfaqrs) My Systems Design knowledge is growing leaps and bounds by watching these and I plan to implement these good designs just after my tests are over. I have worked on several Applications as a backend developer, and always stressed heavily on scalability, flexibility, ACID properties. But, your channel has taught me a lot more good techniques and design concepts.

@gkcs 3 жыл бұрын

Thank you 😁

@MichaelAMomo 2 жыл бұрын

Seriously man, It is a great help

@aakashjapi4394 4 жыл бұрын

Great video! That said, in your descriptions of the database schema, you should mention hotspotting as a justification for certain decisions as well. Namely, a very good reason to not add a "likes" column to posts is that it creates a lot of contention on rows in a single table, especially because single posts can get hundreds of thousands of likes. You arrived at the same conclusion - building tables that allows for writes to avoid contention and thus reads to be aggregations (which can then utilize caching) - but I think focusing on the larger problem of hotspotting motivates your design decisions better.

@rujotheone 4 жыл бұрын

Noob question, please what is hotspotting

@osmanbaskaya7400 Жыл бұрын

@@rujotheone some records are getting queried more than others. the specific instance that contains the record will be much busier than the rest of the system. you're not balancing the load ideally uniformly.

@shubhammehta319 3 жыл бұрын

Hi Gaurav, I love the way you explain things. This video actually sums up all the major components, including the DB structures and High-level architecture. I have one question regarding the design, which is more on low-level design, it would be great if you can create a video on that. Q. If I need to design the data storage in-memory, which data structure we should use to store the posts. likes, follower data, such that we can fulfill the given features efficiently. Thanks,

@anotherdigitalnomad9429 4 жыл бұрын

This was such a good video. Thanks!

@a2zlearninghub67 5 жыл бұрын

Hey gourav sir😊🙏 Nice overview and well explained. You r really great person who share our personal experience. 👍

@gkcs 5 жыл бұрын

Thanks Sandip!

@prakhargurawa 5 жыл бұрын

Awesome explanation ..your videos are really helping ..keep it up bro :) :)

@praveen3123 5 жыл бұрын

Hey Gaurav great videos bro.. Every software engineer should know system designs to build scalable, robust applications.. keep rocking!

@gkcs 5 жыл бұрын

Thank you!

@praveen3123 5 жыл бұрын

@@gkcs by the way now you're a software engineer with at least 3- 4 years of experience by now. Do you still practice algorithms? I'm in this dilemma whether to practice or take it light

@gauthamhonnavara 5 жыл бұрын

@@praveen3123 Never stop learning !

@pankilpanchal1996 5 жыл бұрын

Just a test message weather you reads it or not. BTW very good system design playlist.

@chathurabuddi 3 жыл бұрын

What a wonderful channel!!! just subscribed

@pratiksinha5737 4 жыл бұрын

@gaurav sen Great video......it literally gave me an insight on how to use the theoritical knowledge we have gained as a CS engineer in real Practical Designing

@sahajarora2162 4 жыл бұрын

Hi Gaurav! Very thankful to you for sharing your knowledge with the rest of the world! I have 3 questions about the GATEWAY: 1) Is it a Micro-Service? If not, what exactly is it (i.e what does it contain)? 2) It seems like a single point of failure, looking at the diagram. 3) If we have multiple instances of a Gateway, then would the Load balancers be needed in between Client and the Gateway Service ?

@ZeroInfinityCoder 5 жыл бұрын

@Gaurav i think it should be "postId" in place of "activityId" in "comment table" when you were explaining feature no. 2 as let say we want to find all the comments for a particular post , then we will look into comment table for column postId. correct me if i am wrong

@tharinda 2 жыл бұрын

Much appreciated. Love it ❤🔥

@hellenruthes2056 2 жыл бұрын

I love your videos :) Thanks for sharing!

@jayanthmanklu8642 5 жыл бұрын

Hi Gaurav, Your energy is just unmatched! Audience Request: Please consider doing a video on how would one architect IRCTC Tatkal Booking scenario - with hundreds of thousands of tickets sold in 2 to 3 minutes time duration. Thanks

@gkcs 5 жыл бұрын

I'll try to work on this 😁

@jksharma7 2 жыл бұрын

Thank you sir.... a Great Teacher.

@AlgoHacks 5 жыл бұрын

Awesomely explained 😊 I guess it's just not that simple as it seems after your explanation. Great work is being done behind the scene. Thanks for all the awesome videos.

@gkcs 5 жыл бұрын

Thanks Ashutosh!

@AseshShrestha 2 жыл бұрын

Love you system design videos. Love from Nepal 👍

@amitagnihotri30 5 жыл бұрын

From Designing Tinder to Instagram, in a very short time :D

@gkcs 5 жыл бұрын

Hahaha, just 6 months 😉

@ajayguru2116 4 жыл бұрын

Amazing content!

@vikashdhanabal8984 5 жыл бұрын

hi Gaurav bro, it was amazing .waiting for more such videos

@gkcs 5 жыл бұрын

Thanks!

@siddharthbhola4231 4 жыл бұрын

@GauravSen: Excellent video! Please help me with a question: Can you please explain the reasoning/thought-process behind choosing a Relational database for Users and Feed Schema?? What factors do you consider when taking such a decision? Thanks and Regards

@anastasianaumko923 Жыл бұрын

Very elaborate, thank you! 🤩

@purusottam-tb1sw 4 жыл бұрын

Great explanation man.

@MOhan-ur4ei 5 жыл бұрын

Your English is awesome bro!!

@kennethcarvalho3684 Жыл бұрын

Life is incomplete without a Gkcs design video

@gkcs Жыл бұрын

Hahaha!

@newenglandbarbell4647 4 жыл бұрын

What a talent 👏🙌

@gautamtyagi8846 3 жыл бұрын

thanks, really helpful explanation.

@harisridhar1668 3 жыл бұрын

13:30 Hi Gaurav - thanks for pointing out the need for a load balancer with the snapshot technique stored onto Gateway for network routing when we horizontally scale the server-side. But why is communicating with the load balancer inefficient? Is this to avoid constant network calls ( which are slow ) and to utilize the SS, which can be stored into memory-side on the Gateway application?

@darkwoodmovies 7 ай бұрын

Great video, ty. I'm building an app that does something tangential to this, really helpful for real-world work!

@ivailotenevv 4 жыл бұрын

Awesome, keep up the good work :)

@sankarasubramanian568 Жыл бұрын

Amazing Video, Thanks Gaurav :)

@GilP-BM 4 жыл бұрын

Great video man 💯

@fareidfareid6484 Жыл бұрын

Thanks for your efforts

@JitendraSinghPal 3 жыл бұрын

Nice explanation. Excellent work

@pman-codes 4 жыл бұрын

Hi Gaurav, Greetings. I love your work, I am a subscriber and a frequent liker. However, I find an implicit assumption in your system which considers the Instagram mobile app as thin clients. The process of storing the posts in the cache in the server would result in an unscalable system. I believe the posts are cached in the user's app memory(cache and physical storage), considering that these apps have a considerable chunk of internal storage used. An added proof for this would be if you try to open up Instagram in offline mode you can still see past posts and a toast message which says "couldn't refresh feed". I would like to have the cache on the user's system and then an identifier that is stored in a place where you are storing the cache of the post on the server. (considering the news feed functionality. This can be applied for other uses too. ) Thanks.

@gkcs 4 жыл бұрын

This is a very good point. Thanks for posting 😁

@vikask7246 4 жыл бұрын

@@gkcs can you explain system design for telegram

@dustindiaz 3 жыл бұрын

@@gkcs the posts ARE stored in application cache... but it doesn't invalidate the fact that mobile apps aren't still thin clients. a user can delete an app, or visit from a third party integration (not built by instagram) -- in which case these timeline feeds are still stored in horizontal caches. you wouldn't believe the amount of money instagram/twitter/etc spend on memcache to make this happen.

@adminwadidaw576 3 жыл бұрын

@@dustindiaz so which way do i have to follow? Do i need to cache posts in client side? I confuse in like sectiob , whenever user click like, should the client side make a request?

@dustindiaz 3 жыл бұрын

admin wadidaw caching on the client is helpful when revisiting an application. This way a user can be presented with information immediately. Caching on the server, on the other hand, is necessary for large scale services to deliver things like timelines since raw sql queries based on this system design would cause the system to fall over with just a decent amount of traffic

@RishiRaj-ex9wb 2 жыл бұрын

Great content. Thanks a lot.

@raveenamewani7963 4 жыл бұрын

Hi Gaurav, I really like your videos they are very clear and to the point. I would like you to share in one of your videos how the Amazon Market Place Design will look and work like

@Yan-rv8mi 3 жыл бұрын

7:16 I think we do need a "type" on the Activity table. For example, suppose there's a postID being "123" and a commentID also being "123". Since both postID and commentID can be interpreted as activityID on Activity table, if there a row on activity table with activityID being "123" we don't know it's for the post or for the comment, unless we have a column "type" to distinguish between them.

@jellyrabbits375 2 жыл бұрын

True in this case, it depends on your system though. In case where the id is a UUID, then there won't be a case where postId and commetId would be equal.

@alicebobson2868 4 жыл бұрын

this was very useful, thanks

@raghav4296 5 жыл бұрын

Awesome video series on this channel 💚. One request Gaurav- Could you please share insights on how the video&audio based systems are designed,built and the kind of algorithms/libraries that go into transcoding on scale, as they are computationally intensive tasks. Thanks.

@gkcs 5 жыл бұрын

Thank you! I'll get to designing Netflix/KZfaq in a while. That will be fun! 😁

@abhishekgupta4570 5 жыл бұрын

lot of learning with video and bro one request can u make video on your uber interview about question asked roundwise and that HR round which was pretty tough as you mentioned in video (Got job in uber).

@gkcs 5 жыл бұрын

Thanks Abhishek! I won't be mentioning the questions asked, because we aren't allowed to. "Got hired" will turn to "Got fired". 😝 You can go through the content on the channel, it's more extensive than an interview set 😁

@erichaymer5130 4 жыл бұрын

Fabulous video. Looking forward to more video. Can you elaborate on empirically optimize a ranked feed?

@RaviKumar-jg1hc 5 жыл бұрын

It's amazing... please keep it going...

@gkcs 5 жыл бұрын

On it! 😁

@RaviKumar-jg1hc 5 жыл бұрын

Yes, hahaha...

@vibhorgupta5862 5 жыл бұрын

Great video ! Thanks a lot for these videos, have been very helpful for me. If you get time can you also publish a video on UBER's design.

@AdventureTrekRide 5 жыл бұрын

Awesome video. you are doing just great :)

@gkcs 5 жыл бұрын

Thanks!

@panchmeshaliiapp1771 Жыл бұрын

subscribed .. you are just amaizing

@freedom1225jjy 4 жыл бұрын

also would be nice to have more videos on ER diagram ! thank you！

@TheSaldude1012 5 жыл бұрын

Quality content!!

@gkcs 5 жыл бұрын

Thank you!

@sitharthanmirudhul5411 3 жыл бұрын

Very good explanation bro!

@ikonxTech 3 жыл бұрын

Man you are the best

@sangeetaisaac5520 2 жыл бұрын

Thanks!

@adilsaju 4 жыл бұрын

awesome man!

@dusmannalinwad 3 жыл бұрын

Thanks..! keep it up..!

@ayushiajit3088 2 жыл бұрын

Hi Gaurav, First of all excellent work on the videos :) I have a doubt on the DB selection, so basically what i am understanding is when we need to store information about user we may use Mysql cause of strong relationships etc but since the content [activity] of the user is kind of unstructured wouldn't it be better to use NoSql? by unstructured i mean, we may or may not have caption, may or may not have images, instead can have videos, or comments in that case can be recursively long..Please correct me if im not going in the right direction! Once again awesome work :)

@apoorvbedmutha457 Жыл бұрын

imo it'll be better to have a combination of noSQL and RDBMS for example tables which need to be regularly updated such as no. of likes must be kept in a noSQL DB whereas things like content of a post which are not changed so frequently are better to be stored in RDBMS

@rittwikarudra626 3 жыл бұрын

Very good explanantion Gaurav. The way instagram generates feed today has changed drastically with their new graph api which focuses more on relationships. It would be great to see a video on that.

@gkcs 3 жыл бұрын

I'll look into it :)

@kapilrules 8 ай бұрын

awesome explanation of fan-out scenario

@karanganwani5154 5 жыл бұрын

Hi Gaurav, The video was awesome.

@gkcs 5 жыл бұрын

Thank you!

@sunnyshekhar862 5 жыл бұрын

Take a break from daily routine work. And watch #GauravSen's design videos... You'll get both the idea and chull (read Motivation ) to work on your own projects. Kudos !! Great work Gaurav 🙌👏😊

@vimalvadivelu4708 5 жыл бұрын

Good one! Keep it up!

@gkcs 5 жыл бұрын

Thank you!

@kbagoli29 5 жыл бұрын

Super benificial video

@tendies 2 жыл бұрын

Thanks dude!

@bismeetsingh352 5 жыл бұрын

Simple and easy!

@gkcs 5 жыл бұрын

😁

@urunovtimes5816 3 жыл бұрын

Good and nice explanation. keep going and more improvement

@Eduardo-fk7ft 3 жыл бұрын

I'm building a reddit clone, and your way of designing the news feed gave me a lot of ideas, thank you!!

@blinkkeebs 3 жыл бұрын

me too! reddit clone from ben awad's tutorial? :D

@Eduardo-fk7ft 3 жыл бұрын

@@blinkkeebs No, I'm just going over the reddit site and reverse engineering the features, but I've also changed some things. BTW I'm building it in MEAN stack.

@blinkkeebs 3 жыл бұрын

@@Eduardo-fk7ft Oh! I'm also building reddit clone but I use React and Next.js for the server-side rendering. The db I use is postgresql. I think I should add some extra features for the project and I found this video :)

@Eduardo-fk7ft 3 жыл бұрын

@@blinkkeebsVery nice tech stack!!. Yes, the best way to learn anything is to make it your way or change it, that is something that works best for me. Good luck, and may google be you best friend!! :P

@ayushvrma93 5 жыл бұрын

Hi Gaurav, First of all, a big thanks for all the videos that you are making. I am preparing for my interviews and these videos are helping me a lot. I would request you to please make a video explaining designing of a game, maybe a little complex game such as FIFA, which can give us an idea as to how to implement real-time occurrences. Since all the moves in a game need to be processed in real time it would be different from a system like Instagram as they can afford to have a delay of a few seconds but these games can not. Thanks :)

@gkcs 5 жыл бұрын

Thanks Ayush! I am working on tic tac toe currently. It'll be progressing towards more complicated games soon. 😁

@shishirkumar8335 5 жыл бұрын

Great Video. For use case user-follower scenario, I see technical solution as Graphical DS problem (considering time). Reason is fetching feed both side will become simple and fast. Looks like an interesting problem to analyze/compare various SD approaches against above use case.

@gkcs 5 жыл бұрын

You could try...