You should start a discord one day

I got a bit confused by the Compression & Compression cont. part on 5:10, do I actually need to memorize / know this algorithm? Or can I just let it slide? (also, loving your videos man, thanks!)

Great video thanks for sharing

Starting with this playlist today. Plan on doing 5 videos per day. Lessgoo❤

keeping a in-memory heap for priority queue may be not durable when node make crash recovery. what about if we make index on priority column of the table in sql and it will be stored in disk memory itself?

I love ur jokes WAY more than what u have to say about system design 😂 please go get some sleep

This is it! This is the video you surpassed Stefan

thanks

thanks

How to handle updates, when its reverse of popular users? Like 1 person following a million users.

Awesome as always!

Hey Bro, I'm a big fan of Stefan and their channel. However, your channel actually complements his a lot. He has good depth but doesn't cover all aspect of SD, your videos is very easy to watch and beginner friendly. So don't get discouraged. Hey you have way more followers than his. He said that because he wants bring his value proposition so that he can charge. You're the true philanthropist. Something for you to learn is how he market himself and monetize his channel. Love you bro!

Hi Jordan, thanks for another great video! This one is really interesting and beefy. I have a question about the "hot comments" section. It is clear to me that when there is a new upvote, the CDC triggers the +1 to corresponding nodes in the db. However, how do we know when it is time to expire some upvotes? Keep monitoring the end of each linkedlist is expensive and wasteful, while getting rid of stale upvotes only when new upvotes comes in cause inaccurate stale "hot" comment ranking. Neither of them look good to me.

you inspired me a lot. I'm preparing ML system design interviews, and I found so many gaps in the resources out there that I decided to fill the gaps. I'll start small and hopefully someone will benefit

Hey Jordan, how will you modify the priorities of the existing elements with the way that you have partitioned the db? It seems like we will have to search through all the partitions to find the element and then update its priority!

15:00 when batman visits his grandma

Jordan when you mention snapshot versioning right at the beginning are you talking about vector versioning? Or are you simply referring to how a db may store the previous write along the current write?

Hi Jordan, thanks for the amazing videos, really learned a lot from you. One question here I feel confused is that there are many nodes in the design like task queuing node, scheduler node, for me they are just some services, like task enquing service or scheduleing Service, is there any reason you draw them as "node"? Thank you!

This video really helps with understanding the need for stream processing frameworks.

Chose this playlist over netflix and enjoying this so far!

Top notch! Although I’m not sure the 1B number is accurate. For example just Wikipedia has about 60M websites! Not just trying to find a fault in your video, just trying to reason about the design. With such a high number, it might not be possible to crawl all of Wikipedia from just 1 host..

About final architecture diagram: Where do you store the actual data about which URLs were processed, or failed to process, and which S3 results links go with each URL? I'd think there needs to be some datastore to maintain all this. Also, if a crawler dies, how do you recover the robots.txt information to prevent any new crawler from breaking the rate limits?

The anti-entropy mentioned in 16:54 doesn't seem to be actual anti-entropy... Anti-entropy implies that data on each node should almost always be in sync, but occasionally goes out of sync - so we detect out-of-sync state with a "cheap" detection protocol, and only need to occasionally follow that up and fix mis-aligned state using an expensive operation. In this case, the data on each node is is almost always way out of sync... so anti-entropy wouldn't be what we use. We'd literally just share state without any cheap detection mechanisms. So maybe I'd call it "gossip-based sharing of hashed state" instead. Because what you are really implying (I think) is that you will share hashed values between the nodes using an epidemic protocol versus a broadcast or centralized. This is orthogonal to the entropy comment which isn't really what is going on here.

12%4 should be 0 rather than 2

Could you also discuss a raffle system, in any system design video like this later? Its something nike does to handle bots. I mean a third party service does it for them, but anyway seems alright.

How does receiving of message happens, once it reaches to Flink node. Is someone subscribing to this Flink node? You mentioned load balancer receiving the message, then isn't that going to cause thundering herd problem?

Great video! In the final design, where is the rate limiting logic implemented? Is it in between the load balancer and the redis database? Or is it implemented on the backend services? Or is it just for us to decide based on the constraints?

Hi Jordan, since you have talked about showing messages of particular chat to the user. If we have sorting of the messages done on the server-side, then instead of returning all the messages and sorting on the device, we could have lazy-loading of the messages. Server can send paginated kind of response for all the messages and load as user scrolls. Does it makes sense?

I dont think you can do pre-materalize for the idempotency. If a user click the pay button twice, the second pay request will ask your "pre-materialize key service" for a new idempotent key. Now the problem goes back to the original one, how can you generate a idempotent key for the request in the first place.

Had me in the first half 😭

The calculation of runtime at 13:40 is wrong. Runtime is O(log(n) + m) where is n is total words in dictionary, and m is total words matched by the query

Bro looking nice and chubby today, Jkjk nice video 😂😅

Hi Jordan, can you do design for Price Drop Tracker (like camelcamelcamel), Privacy & Visibility Controls, Designing Internationalization. Those are popular questions recently and almost no materials about it

Great Video please keep it up

Oh ! I thought “chubby” is self criticism but it’s a system name lol

I like chubby

Hi Jordan thanks for the video - I feel like we haven't touched on how the producer to broker portion of streaming is fault tolerant? As in what if the producer goes down before receiving acknowledgement from the broker, or if the broker goes down. Or any failure scenario in general - how can we ensure the message was processed?

Pretty solid video. Really loved the optimisation you discussed.

At 22:52, you showed a CDC from "user-following" table (so deriving a table from an already derived table). But your final architecture diagram does not have this. That whole section of final architecture is a bit hard to understand and relate to the rest of the video.

good stuff dad

Thanks Barry Keoghan for taking time out of your busy acting job to teach us LLD

How flink will send the messages to the load balancer or the server ?? Aren't we suppose to use kafka here between the flink and the load balance ?

What device do you use for the hand drawing?

Thanks for the great content Jordan. I have a thought and I want your input: Since your text CRDT basically eliminates conflict, you don't need a versioned vector for resolving conflict again. But I think we can still use it for: 1. avoid unnecessarily merging the CRDTs (i.e. if two nodes have the same VV, then they don't need to merge, or if one vector is strictly smaller than the other, then it can just simply discard itself and the other's CRDT values) 2. use the VV to filter out the unnecessary writes. (i think you covered this implicitly) 3. we use VV to create a partial order and thus achieve causality, although I can't be sure whether we need causality? (conflict free should already promise convergence, we might not care about how we eventually get there, except that we want document versioning?)

Thanks a lot for great contents and mainly to continue your dedication to keep producing these. Quick question: you mentioned Riak can be used, so that if conflict occur, we will assign one the next available sequence. But, at that point of time, we might have already provided the short urls to both the users, that will still be same issue as Cassandra? Am I missing anything here to use Riak?

Thanks for the video!! I have one question, why don't we partition the chat servers by combination of video-id/user-id or even just by user-id?, this way we won't have an overloaded chat server for popular streams

Modern system more CPU bound instead of network bound -> not sure I understand it correctly. If it is about latency, network is definately taking more time. QPS wise, CPU bound can be solved by adding more nodes, but network bandwidth will be just like that much? (open to discuss, I don't have any experience on storage on my own..)

I am curious if it is common to use two type of DB in real use case (of course for big company as KZfaq it's worth, but considering we are designing for a team / an org tech work), maybe compared with adopt cassandra, optimize based on mySQL will be more like a real case?

Excited for you! And excited for what comes next! I will await those deep dives so I can CRUSH my next sys design interview (my one motivation to consume what you're planning next is that one interview where I failed where as a junior SWE I was asked to reason along the lines of a research paper -.- yeah im petty i just wanna get back) Good luck!!! And we'll see you again!!! :)

if we use video id + ts as index for comment, will it be case that some comment are posted at the exact the same ts?