Drop some studying and leetcode tips fam. How long was the prep?

@@Ms452123 Probably like 3 months cumulatively of prep, ran through sys designs video on here, grokking, and Neetcode. Also got Leetcode premium and did company tagged questions.

Dude holy shit congrats, let's fucking goooo! May need a referral there one day haha

@@ragnawrawk769 Damn I am doing the exact thing bro, since April though. I have a newGrad Bloomberg interview in August. Still don't feel fully confident. Hopefully in two more month of consistency I will be good, if not for Bloomberg than any other company, because I am desperado lol. I dont have a CS deegree.

@@Ms452123 fyi not to have you stop watching my channel but I think for that you may wanna mostly prep by leetcoding

I appreciate it!!

I think that's reasonable! In reality this design probably would use a dedicated notification service, the one thing to note though is that I feel like if you say that in an interview they'd probably just ask "how do you implement that" haha.

🥰🥰

Nice!

As I come across new concepts that I feel deserve a dedicated video, will do! I appreciate the positive feedback and thanks for viewing, hope this all helps!

That's fair enough, though I should note that with signed URLs I'm not overly concerned with letting clients upload to S3 directly.

My pleasure, considering changing my channel name to something more serious like Jordan gets no bitches

Thanks!! I tried to improve upon this part of my design in the 2.0 version of this video. I think that for popular files we should do what we do in twitter and effectively have a "popular file" cache.

While I imagine that this is possible, it seems like a poor idea just because there's tons of latency induced with a distributed lock. I think you'd be better off just evaluating everything based on the specs of the database :)

Seems reasonable to me!

Yeah I actually am proposing to just upload straight from the client, should in theory save a considerable amount of time. I see your point on Cassandra, but for the sake of avoiding write conflicts I do think that something like MySQL makes the most sense.

You could be right - it's all about how you estimate the capacity really. But that being said I think there may be some org wide docs that 100,000+ people may listen to. Perhaps for those you could just use a pull based model.

I touch upon this a bit but HDFS tends to be more expensive to run as it requires you having your own cluster and scaling it on your own, I think S3 is cheaper so if you just need to store static files it tends to be better

I appreciate it! Maybe the best way to structure the DB here is something like: fileId: 1, chunkOrderNumber: 1, version: 1, hash: abc fileId: 1, chunkOrderNumber: 1, version: 1, hash: def fileId: 1, chunkOrderNumber: 2, version: 1, hash: ghi Then query for all of the files with fileId 1, and groupBy chunkOrderNumber to get the hash with the maximum version - sorry if that wasn't clear during the video!

To be honest, I'm not sure why you feel that the splitting logic would have to be coded per device, there's more or less existing libraries that can do this type of thing for us! Secondly, at least assuming we're using Kafka, that uses long polling under the hood, but basically yeah we'd have to be using some sort of real time communication protocol. Thanks for watching!

yep

I think we agree here, hence the use of userId and fileId

Some marketing analytics team, I don't really even know the specifics yet lol

Definitely don't think you need websockets for it as nothing about this needs to be particularly realtime - look up FTP on wikipedia :)

I think that's a very reasonable optimization as opposed to having to get all incremental changes.

1) I'm using the fileId as the primary key, not the chunk hash. Allows us to keep all chunks for a file adjacent to each other in memory. Using a hash allows us to perform much of the computation on our local machine and then send far less data to the db. 2) We could definitely have a leaderless replication setup, however the penalty of having write conflicts here is pretty large. We generally want to make sure that people are seeing the newest version of a file when they read it. Hope this makes sense!

​@@jordanhasnolife5163 1) Sorry, I meant chunkHash is the PK of the Chunks table. For that, why not use a normal int PK? With your design, we're asynchronously sending actual file contents data to S3 which doesn't require a hash. The only time we need it is to update our metadata db, and the only reason we need it is to know if a file has changed. Aren't we doing that in the client before anyways? Also thanks, you're the man!

@@SmokinBear199 Yeah I think this is a fair point, assuming you're capable of generating the hash on your client machine :)

You can just have a field called chunkNumber - shouldn't be too hard I don't think

While you are correct that either way every user will have to receive the change, offloading the change to a queue based on fileId as opposed to many queues based on userId will (at least in my opinion) reduce the amount of work being done by our central server and shift the load to the message queues which are then responsible for delivering messages to clients. I think you can make an argument for both, as you also wouldn't want an invididual client to have to maintain too many websocket connections.

​@@jordanhasnolife5163 Seems we're talking about two different things. Each user will have to have their own "channel" (or ConsumerGroup in Kafka system's term) to receive their copy of the message, AND they will also have to receive ALL files updates they subscribe to. When you say "partition by fileid" I take it means one user may need to subscribe to multiple message queues in which certain files they need to subscribe as topic, in which case, it's not that different from grokking, you still will have ALL users subscribing to these same queues and same fileid topic as separate ConsumerGroups. "Partition by userId" would be weird but idt that's what grokking means, essentially, each file change results in change stream published to whichever queue this file supposed to go to, and then you have different consumergroups (or kinesis fanout streams) pick those changes up. Grokking brushes past it by saying "separate response queues" but that could just be talking about the consumer fanout part. Other resources sometimes build upon that and say "broadcasting" which adds to further confusion. Is that a fair understanding? This understanding also assumes we're not setting one queue/stream per file either, that seems would be very wasteful? If you talked about a user maintaining 200 websocket connections it would appear you think we will have literally every file in a separate queue? But on the other hand, if we don't do that, then partitioning and distributing which files go into which queue and how that makes consumers listening to as minimal queues as possible, and not listening to queues that have tons of files they are not interested in, also seems tricky?

First of all, that's a great idea to come up with your own solution before watching! What you're describing is more like a realtime text editing platform, and you've basically just spoken about operational transform, where all edits to a file are routed to a given server and then ordered and basically merged together using some sort of algorithm. It's definitely possible, but it assumes that one server can also handle all the load for a given doc, and also I think this strategy may be over engineering a bit for this problem, but very relevant for something like Google Docs, which I will eventually cover!

There's also the question of what happens to clients who have edited a doc locally and then the websocket pushes a new version of the document to them? Do their local changes just get thrown out? Do they have to perform a merge operation?

Gotcha, makes sense. Thanks for the feedback! I think I got ahead of myself and started designing without knowing all the requirements first. Fatal mistake! 😵 Now that I think of it, I would be shocked if I was ever asked to design a real-time text editing platform in a system design interview but if I am I can hit them with something like this. Thanks for the videos, man. This channel has helped me a lot!

@@neek6327 haha all good glad to help and happy to discuss in more depth! I have heard of people asking for Google docs before so I'll do that one soon enough! Plus it's a cool topic

Kafka

@@jordanhasnolife5163 I did not expect you would reply this fast, thanks again. To my understanding, each partition will contain files of different users. How do you handle security here (prevent a client from reading file changes which it does not have permissions to)?

@@iambao1940 I think something like encryption might solve this

@@jordanhasnolife5163 by using encryption, you mean the client can still receive the file changes which it is not expected to receive (files of other users) but cannot parse the content. But I think it wastes unnecessary high network bandwidth. I am expecting some kind of filtering with which the client only receives the files in its permissions.

I think there's a misunderstanding between us here. It's not actually necessarily a queue per file, but rather the fileId is partitioned by hash range into queues. That being said, I still don't see why that would limit unique files - we can have many partitioned queues, and any server can push to any queue. Can you elaborate a bit more please on why there is a 65k limit? If anything, it is a limit of 65k clients that can connect to a given queue server, which should be ok assuming there aren't more than 65k people concurrently trying to access a document, and even if there were the queues can be replicated.

@@jordanhasnolife5163 ah I got you, and the 65k limit is per client ip to server port. So one client (on one ip, clients can have multiple ips!) can have 65k connections on one server port.

In the above client #2 with different ip can have its own 65k connections to the same server at the same time

@@markblum4854 understood, just confused how that limits the entire service to 65k files, I appreciate the input!

​@@jordanhasnolife5163 Oh because you said if a user have 200 files they will have 200 socket connections, maybe I guess what you meant is "at most" you will 200 connections but in reality you can have fewer because your files should be designed in a way to go into one queue (then pushing that to extreme, it will be as if that queue IS catered just for that user). I wonder in real world is it feasible to just publish one file per queue for all the files in the world on google drive or dropbox, or that's actually not an issue.

Yeah I really used to talk slower in the older videos for whatever reasom

His hair was gross

Partitioning partitioning partitioning

@@jordanhasnolife5163 splitting the file doesn't really work, does it? If I'm uploading a 10 GB file and I have 9 GB left and the chunks are coming in 4 MB at a time... do you just backdelete the other chunks?