You should leave Google for Twitter

tweet this video to Elon , he might make you CEO, he is weird like that.

@@sushantbhargav4652 lol maybe i should

Yes hire then next day fire

@@NeetCode you should look at your website, I tried to go pro but for some reason the google api won’t let me sign up. I don’t know if I’m the only one the problem or if it is general.

10 Years and you barely did system design? Typically getting up in seniority means having to take a higher level approach to problems and leaving the implementation to juniors

@@garlicpress6121 I feel like the web based software has skewed everyones perception and it makes people think that this is the only kind of sodtware dev in the work. There are so many other domains which would never need to know this sort of stuff for interviews or even for their work. For example, someone working on low level programming for drivers, or OS level sofware or desktop applications.

Thank you, more to come!

@@NeetCode I think he meant on your youtube channel haha..

@@sanskarkaazi3830 obviously what else could her mean?

@@indiging8330 neetcode has premium courses on his website as well so not there but here.. you get what i mean?

Can you talk about Pinterest, or someone link some available content.

Glad it was helpful!

Musk will hire him

Its because Musk tweeted the HLD of twitter on twitter. You can see that in the thumbnail of this video too

no, with such test you filter already for ex-twitter employees. That would be fine if you build a social network, but you'd miss out on all the all the brilliant devs who for example designed large e-commerce or data-pipeline architectures, because that requires a very different approach.

I'm sorry to hear that, wish you the best - it's only a matter of time!!!

me too😂😂

It will also be a case study on if these software companies are truly over staffed or not. If Twitter survives after laying off so many people it may inspire other companies to consider down staffing

@@KennethBoneth I think the main issue with scaling down on employees is that the remaining employees will essentially have to monitor and handle the same amount of work as before scaling down, which will cause additional stress and probably a less than healthy work life balance.

Not really, Tesla and SpaceX both are well known for the horrendous work environment. So it depends on the management and the owner of the company in this case.

@@bryanyang7626 that's true, might not work well with other companies once people start realizing their lives are worth more than slaving away

@@Mattarii That is true if you were properly staffed to begin with. If twitter is as overstaffed as many people believe, then a large chunk of employees are effectively doing nothing. IF twitter goes from properly staffed to understaffed, you are correct. If twitter is going from overstaffed to properly staffed, then that won't happen.

how is this a quality content?

​@@umarqureshi8499what's wrong with it?

Yes, that seems like a big oversight. Each shard will have a subset of a users followees, so the proposed user id as a shard key really doesn't do anything for us.

Yeah I felt like I was missing something when he said sharding and scrolled down to the comments to confirm

Just paused at that part, seems incorrect. The best sharding I think may be tweet id (assuming using chronological IDs like snowflake) as people are generally accessing the latest tweets so can grab them in a single request if it misses cache

@@salient244 yeah, but still you'd need to store the friends relationships somehow and you'd get into the sharing issue when it scales up

You've got it wrong. The idea is to have all the _followers_ of the user in one shard. This way, when the user posts a tweet, you would get all their followers ids from one shard with one query. Then you'd use this list of ids, to update their respective feeds with the tweet. When the user request their feed, they get it pre-computed from the cache, not built on-the-fly.

So...use the average Twitterer's tweets as load dampening. They should do that. It will make Twitter even less popular.

I’d try checking their engineering blog for leads.

That more or less is I think what he described for his feed cache description. But it doesn't solve the problem he brings up where we don't want to update all the followers' feed cache whenever a popular user posts a tweet. Also, I don't know how to do it, but when you say "on update", I'm assuming that whenever a person posts a tweet, all the users following that person gets "updated". In that case, then only thing that needs to be changed is inserting that new tweet into the feed (and probably popping out whatever oldest or least important tweet that is in the feed that this new tweet will replace). In that case, I don't think retrieving and merging all the relevant tweets each time there is an "update" makes sense. I think that's why he brought up pub/sub. So it's just a queue where whenever a new one comes the least important one gets popped out.

​@@marspark6351 Maybe it's possible to determine a "popular" user and when those users create a tweet, only cache that tweet instead of allowing a message to go through the pub/sub when they post a tweet.

Probably, having NoSQL KV-store with such massive reads you'd have to deal with its sharding anyways. Don't think you'd just set up Cassandra and start throwing in nodes to the cluster mindlessly. So, author, choosing SQL DB, just makes that logic explicit.

I was left with the same impression. I don't see how this sharding could work

do you mean by passing user ids along with the request implies that the auth token is stored in db? because I don't see him mention it explicitly in the video where to store auth tokens. Also out of curiosity where do we store the auth tokens then?

@@zhenghaohe4727 we don't store them, we validate them against our secrets

DDIA is the most comprehensive resource (assuming you have at least some experience). Also, most companies (including twitter) release blog posts and white papers about technical challenges they faced and how they overcame them. I think many beginners miss these, but they are an extremely valuable and free resource, which is why they are commonly referenced by system design textbooks.

Thanks!!

@@NeetCode Is there a central url where you find those blog posts or do you just google them?

A good place to start is by learning the classic OOP design patterns. It's less about the OOP and more about the patterns.

Yep, but there is no requirement in this case to be able to request tweet by id directly without knowing the author of the tweet.

😅

Before returning the feed, the app server would check if the user follows any celebrity (one query to follow table). Then get the tweets of the celebrities which user follows from the cache, and inject them into the feed based on the timestamp. This approach has significant downsides like increasing latency for all users, so I believe this problem is addressed differently in real world.

That's correct, I meant that while NoSQL is easier to scale (automatically or by specifying a shard key), we can still scale relational DBs via sharding.

Nice catch boss

I would believe this has less to do with whether it's SQL or noSQL, but probably more to do with that Redis makes better use of RAM than mysql. Don't take my word tho. Just a possible assumption

If we wanted all the people that user1 follows (in order to generate their news feed) we could run a query like: SELECT * FROM followers WHERE followers.followerId = user1 Notice we are filtering by followerId.

Whenever we create the newsfeed we want to populate it with tweets of people that a user follows(followee). So when we index the follower, we can query the followees relatively quickly, meaning that we can get all the people that a user follows, which makes it easier to create their newsfeed.

I understood it as: when the timeline is loaded and we need to populate tweets, you're going to be querying for tweets based on the follower of that tweet as opposed to the person being followed (followee). So if we say the user loading the timeline is the current_user a query (in a simplified world) would be like: SELECT tweets.content FROM tweets WHERE tweet.follower_id = current_user.id Note how our WHERE clause is on follower_id of that tweet and NOT the person who wrote that tweet (followee).

I see. Thank you all for explaining this!

Paint3d