I think you can tell that Mark has had a lot of experience as he's able to navigate easily even topics that he declares that he doesn't master and he's also an excellent communicator. Those are very valuable soft skills and not many people think about them. Also, I loved his enthusiasm at the end. However, I imagine that Mark hasn't had a lot of changes to build systems in the recent years as there are some slight technical issues with what he proposed and, since this channel probably attracts many beginners, I'll go over them and try to come up with better alternatives than what Mark proposed. Hopefully it will be an useful exercise! First, the API is a bit wrong. Normally you put the version before the resource name, so instead of `/messages/v1`, it would be `v1/messages`. However, what's more interesting here is the way we model our domain, as it's not just about users and messages and it's mainly about conversations: when you open the application you do not see messages, you see a list of conversations and the messages associated with the latest conversation, when you want to send somebody a message, it is part of a conversation and only one conversation can happen between two users, plus there are features like blocking, notifications, etc that are associated with conversations, not individual messages. So you'd work with something like `POST conversations/conversationId/messages` and then you don't need to add information about the recipient as we already know the conversation ID from the URL (and also scales nicely when thinking about groups). Also, you most likely don't want to mark as read each message individually and instead you'd probably want to mark the conversation as read until a specific message ID (imagine a chat with 500 unread messages, it would be horrible to generate 500 requests just to mark a conversation as read). As for the database, I think Mark kind of rushed to the NoSQL solutions. You can easily scale relational databases like MySQL (Facebook, Github, Shopify are all heavy MySQL users) or PostgreSQL and there are managed offers for both databases. The real question is if you have relational data and you can take advantage of those relationships. And I think we do have relational data: a user has many conversations and a conversation belongs to two participants and a conversation has many messages. Speaking of messages, how do we handle the sent/delivered/read status of a message? By default, a message has the status "sent", otherwise it wouldn't be in our database. Whenever the other participant's application pulls conversation data (either in the background or when the user opens the app), it sends back the ID of the last message received, so all the messages up until that point are marked as "delivered". Whenever the user opens the conversation, the app sends back the ID of the last message that appeared in the viewport, so all messages up until that point are marked as "read". This is more efficient because you can do bulk operations on messages and the logic that determines the limit of those bulk operations is offloaded to each individual device instead of a process that loops over billions of messages. There are some finer points, like how does the application know that a new message has arrived. It could use the push notification system from Apple or Google, but there are no guaranteed deliveries. It could poll an endpoint, but that would generate massive load. It could also open a persistent connection (like Websockets) and receive messages that way, but you will need thousands of machines if you're targeting billions of people. And many others...🙂

I find it sad that there was no talk about websockets or any other technology that would enable a real-time chat experience. How are two users gonna chat? we designed an API to post messages and an API to get unread messages. Are we supposed to poll the server every second to get the unread messages? how is that gonna work for telegram that has 1 BIllion users and 15Billion messages daily. And what was that logic of looping through unread messages about? how it that possible with these numbers. I think the good part of this interview was the begging, since the estimations were done great though I find it disappointing that those estimations did not really play a role for the next 40 minutes of the interview. We did not use any of our estimations during the actual designing process.

This channel is great, but from what I have seen so far it could improve a little bit by making the interviewer challenge the candidate a bit more; this would be a more real-life scenario than just "agreeing" with all the candidate is doing. i.e. Not all Mark's decisions are ultimately flawless and there are tradeoffs to consider that should be mentioned. Great work, though!

Looping over messages in database with 10B messages a day? Seriosly?

Did not talk about how can users chat, nothing about things like websockets, polling, sse, eventual consistency etc. Should everything be tranferred with HTTP(S) or there is something more efficient? Should all the chatting go through the API servers or we can implement direct communication between the (two) users? I would also not suggest specific technologies (like DynamoDB), but I would rather say something like "I would pick a database that has this and that properties, so that we can address this and that problem". In general, I expected more, especially from a person that served so much time at G.

I would really have to say that this was a great interview! I just have one nit on the design. I really think a stream/MQ/queue approach to dealing with a message distribution would be more appropriate in this case. "Looping over the database" is a huge waste of resources/bandwidth/etc and a red flag IMO in addition to data consistency and state issues with multiple processors. I can only assume that Mark expects to query an index repeatedly which could also be expensive based on the required latency. Processing message queues scales quite easily and decreases the latency between receiving and routing the message. (ex-G 10y)

Mark is so amazing at going through the design in such a simplified but detailed way. I decided to use his approach in an interview and it worked really well for me thank you Mark.

the interviewee is having so much clarity in this thoughts he's amazing crisp clear i love the way how he sticks to just limited details than talking or going through unlimited details in this video, which would lead us to confusion. that greatly shows his experience too

It's a funny thing that ex-Googler tells more about AWS than Google solutions 😂

Why do we need a message distributor? When the api server queries and creates a new message entry, it can make an extra query to update the unread_message_ids of the receiver. Doing so can help to get rip of the message distributor.

mark's accent is very native and comfortable, this is a good material for practice my spoken english

I know this is for people training their interview, but boy is this content gold for system architects and people that want to build stuff like apps or startups. Just found out about the channel, 10/10

thank you for the content it would be great if the interviewer can ask some questions instead of only praising the candidate: like where do the message text sit?, any caching of messages, pull vs push stretegy of delivery of messages and their status etc

thta really awesome. I learnt a lot. One thing puzzled the distributor and from there many question coming. 1. should we really have a massive table on messages? if yes how sharding/hashtable should be implemented? how users and messaging will behandled if both remainsin different box? Otherwise Little elaboration on table structure and sharding would be of great help/ 2. what if considering KAFKA kind of storage system and remove all delivered, hold copies not delivered with some count and day basis. This will eliniminate storage requirement.

Why did we go with HTTP REST over a web socket?

Couldn't the message distributer be replaced with an event driven solution like kafka/kinesis?

Would have been nice to compare this approach to an Append-only-log structure like Kafka as a store - I suspect that's where we would have ended up if group messages were part of the design. I think not handling groups was a miss here. I would expect a senior/staff/principle candidate be able to talk about the added scaling needs of group chats.

Great interview! This channel is quite underrated. Keep up the great work! 👏👏👏

Shouldn't we use WebSocket instead of Rest? Why to establish connection for each msg? Also Server can't send request if rest is used

Great job! Enjoyed the way you are doing it❤

His design really reflected his lack of understanding of how front end works

A good work by IGotAnOffer , detailed explanation on system design questions. All the best!

For the messages not yet received due to a receiver not being available, wouldn't a pub/sub system be a lot more efficient than constantly looping over the unreceived messages? I.e., message discovers it can't be received, so it instead subscribes to the user-just-became-available event and gets stored in a dedicated DB while it waits. When the user logs in, the event fires and the message gets sent.

I have a senior systems interview tonight. I've probably watched 100 hours of breakdowns and mock interviews in the last 2 weeks. Wish me luck.

So is each client polling the server every 2s? Resulting in a db query each time?? To check unread messages

So we're not going to discuss the main selling point of Telegram: Security? End to end encryption ring a bell?

Nice video, thanks a lot. However, I want to highlight one point that could be useful for those preparing for this type of interview. Do not add more functionality to your system that was discussed in the "Functional Requirement" step. It may play against you in two ways: 1) It could be one of the red flags for the interviewer. 2) You can implement this functionality not optimal and this will also be a red flag for the interviewer. In this interview, Mark added to his system the ability to keep track of the message's status. And this was done, in my opinion, not optimal at all. Nevertheless, thanks for the video.

The first part started good, but the API part is all wrong Those are four Endpoints of ONE API of one service. Also, why would you go so deeply into the implementation of Endpoints for a high level design portion of the system?

Where is a Text in the message table?) I guess it was lost

Are we shifting the values from unread_messages_id's to read_message_ids once user reads the message ? Please clarify more, if yes then how ?

19:54 yes we did put a number there :)

as a senior eng you shouldn't ask for scale of the product you are designing, you should make educated guesses and have the interviewer confirm it, better yet do a solution for small scale and then show the way to scale up for bigger traffic

a google em uses job runners to distribute messages for a 500k m/s application?

Why there is a need of message distributor ? At the time of receiving messages, one can just update the unread_messages_list of the receipient. Why receipent has to be online for DB update ?

Did we actually need a message distributor? Why not add the messge to the recipient's unread messages the moment it gets posted by the sender?

That would be great if Mark makes course on system design, I would definitely purchase it.

This solution would get easily rejected in a real-world interview. No discussions about how clients interact with the backend - how do we ensure a real-time chat experience? WebSockets, long-polling, etc. The scheduler kind of job that keeps checking databases for undelivered messages etc. doesn't make sense. The interviewer doesn't challenge the interviewee enough.

Not one comment about security? Encryption? Omg.

I like how Mark handles this, but I see clearly from his videos that he's not really been hands in implementation. His approach to API and design on Telegram is really flawed. Sad to see the interviewer also didn't correct him.

I was not satisfied with the idea of how he was storing the read/unread messages.

Any idea about the cool tool Mark is using for drawing?

Hey! you are doing a great work, helped me a lot and I learned alot, just wanted to give you a suggestion, Can you ask more cross questions in these type of interviews. :)

Appreciate his experience and expertise but he designed Telegram without even talking about real-time messaging! How can an interviewer accept this basic and simplified system design for a platform like Telegram? Chat app without a real-time client-server connection is not a real-world system!

No need for websockets, redis, kafka? :o

Shouldn't we sketch out the data model before selecting the db?

12:52 [Candidate] Let me put some.. [My brain: let me put some dirt in your eye]

Thanks for the video. Best link is me: simple and free!

what seems appealing to me is that interviewer does not ask question during the explanation

The better architecture would be like below. This will fix all the For sending Mobile --> Message Server --> Kafka --> Consumer --> Databse For receiving Mobile

Which diagram making software is being used here? Can anybody please tell me the game? Thanks.

could you store messages in s3? this way you can store images there as well.

This is just for learning purposes, if you bring this design to the interview process, you will get good amount of questions from the interviewer 😅

Thank you so much! This is an excellent resource for everyone to enhance their knowledge and delve into deeper thinking. I have a question: you mentioned that you were storing sent, read, and unread messages under Users. Considering that the number of messages will likely grow over time, there might be a concern about exceeding the storage limit for a single document (or row) in databases like MongoDB, where the maximum size is 16 MB. Couldn't this potentially lead to issues in the future? How do think to resolve this?

Need more of these mock interviews

I wonder what happened to the other videos of Hong Lu, did they get deleted?

24:00 Cassandra is not a good fit for this usecase. The trade-offs are completely missing.

At about minute 40:00 Tom struggling staying awake 😂

Awesome, thank you

What program are using in these interviews?

It's only me or others that find you two look alike.

great information

Insightful interview but why the interviewer is silent all the time? There are millions of questions and drill downs that he should ask but he just sits and nods. I'm sorry but It would be 100 times more useful to make an interview with a real interviewer not with a nodding head. You won't see such an interviewer irl and if you would run from this company.

Golden content

Look at the draw here.. Is this a real Googler?

Just out of my curiosity, is there a requirement to store the messages on the server specifically? I was thinking of using apache kafka as the backend, is there any challenges using a message queue for such scenarios?

4:10 he mentioned if its okay to use APIs as a point of entry. What other points of entry could there be? Not very experienced, but I thought that an app needs an API to connect things around, kinda like a middleman. What other options are there?

thanks very usefull

How/Why did we assume that one server can handle only 10k requests/second?

the interviewer was set to responds yes to every decision the interviewee made -_-

Learn about the system design of Google drive here : kzfaq.info/get/bejne/h66ipNZzxJnUeoE.html

So it's not a real interview

this was epic

Can’t really start with how this design system is bad

I certainly liked that interview, but that design... Let's be honest, drill down was absolute mess. You just can't store messages inside user on that scale. Dot.

great video.

45:30

I expected better! :/

Get affordable, 1-to-1 expert coaching to ace your system design interview: igotanoffer.com/en/interview-coaching/type/tech-interview?KZfaq&

Message system is good but last time I was on system design interview they ask me to design air traffic control software