excellent work of animation, design and understanding. I just want to add a small point here. this is just plain scheduling (decades old) and load balancing is one application of scheduling. The topic of scheduling itself has been well researched in CS research community. The actual researchers here are to be praised more.

@@mpty2022for sure! I want to write more broadly about the topic of scheduling soon, there’s a tonne of literature to absorb first though 😅

@@samrosewho oh man you did a great work, i feel like an a** now

@@mpty2022 it’s alright, we’re all standing on the shoulders of giants. Important to remember that 😄

Great article! Thanks!

Really well done. I'm a visual learner so this kind of style is very helpful.

Hey I have a question about your post. Because you mentioned that in terms of dropped requests, PEWMA starts better than LC, but that eventually it starts performing worse. But I don't really understand why that would be the case. As in, I understand why that's the case for how you explained PEWMA works, but the fix seems (to me at least), trivially simple. You start with PEWMA, but the load balancer also knows which servers have their queues maxed out, so once you know that all your fast servers have their queues maxed out, you start using the slowest servers again, effectively falling back to the LC algorithm. And of course, then once traffic slows down a bit and your servers have some breathing room again, you switch back to the pure PEWMA algorithm. Isn't it that easy? Am I missing something here? Edit: oh finally, I absolutely loved all the animations and especially the playground at the end where you could tinker with the parameters. I was wondering if it would be not too much to ask, if you could add to the playground an automatically updating graph showing the 95% percentile of latency of the last 60 seconds of requests and / or the number of dropped requests of the last 60 seconds. Because I noticed that once you push the algorithms close enough to their limits, it's actually quite hard to get an accurate feel of their performance simply by eyeballing it.

I've been building servers and full stack apps and always hated the issues I ran into with using Load Balancers. I appreciate the very good breakdown of this. This is now part of my reference material to share with others.

@@NicholasMaietta really glad you enjoyed it! ❤️

LMAO

Ops/infra has some of the coolest computer science, but also lots of coverage of it is quite dry and intimidating. I’m having a blast trying to bring the ideas to life and make them less scary!

All devs appreciate ops, but javascript devs.

@@charliesta.abc123very accurate

Every dev appreciates ops. You must not be a dev.

Oh damn… I didn’t realise this! Makes total sense though, and now I feel silly 😅

Is that because as you move into higher and higher frequencies the number of samples to represent the signal decreases? If I remember right the simplest fir low-pass is just the average of n and n-1. So as the number of samples per frequency decreases the difference between the amplitudes of adjacent samples becomes much more significant. I never thought about it as an exponentially weighted moving average though, probably because I'm a dumb novice programmer and huge audio nerd lol

@@roycrippen9617 One more fun fact about EMA: it is also used in LLM training. I don't think you're dumb, there is just a crap ton of knowledge in this field. EMA (exponential moving average) is an IIR filter that can be derived by discretizing resistor capacitor (RC) analog low pass filter. The analogy that clicked for me was that the original filter pulls the moving average towards itself with a rubber band. Low frequencies have enough time to change the position of the average. While high frequencies kind of just wiggle it a little because they change too fast to attract the average. This analogy is also reasonably mathematically accurate.

@@jst1977first thing i thought of was the adam optimizer! funny how these things pop up in places you'd least except

Many technical term like load balancing is not a "simple" topic once we start to think about how they can work. That's why I hate the modern interview process for non-senior above involves system design questions (that suggests and encourage the interviewees to use those terms as an unspoken rule), where random junior/mid devs keep using those term to randomly brag about we can horizontally scale our system to handle billions requests in random design for non-sense cases without tradeoff and penalty. Everyone pretends they know that during the interview but in fact most (includes the interviewers) don't without a long research. In reality, often the person like me (not try to brag I'm good dev at all, just average more pragmatic one) who can't pass those random questions, will have to implement the similar stuffs in real servers. Those who brags they know it during the interview? All their theoretical assumption and imagination without the true past experiences literally failed at the very first step.

There are strategies to avoid the kind of behaviours you’re talking about, but none of them do as well as round robin at minimising loss. I wouldn’t go as far as saying that it optimises for availability, but it does a good job at avoiding pathological behaviour. It’s also wonderfully simple and good enough for 99% of use-cases 👌

Where are you seeing ads on the post?

@@samrosewho I’m not, I just thought the concept of doing that would be funny. I’ll edit my comment cause I think it wasn’t clear that this was a joke

@@miscbits you had me worried 😅 thought they may have slipped in somehow. Thanks for the edit!

Said thank you on Twitter but will say it here too: thank you 🙏

The first step is getting your application ready for horizontal scaling. Does it tolerate reordered requests? Do you also need to scale your database? If so, how? How do you handle race conditions? There's probably a metric fuckton of additional important aspects I've omitted here (not a dev, I'm an ops guy). If your application is ready, you can think about implementing load balancing. There's great FOSS load balancing software available, most notably HAproxy and nginx. Both work great and have their own idiosyncrasies, but you're probably already familiar with nginx because it's also a great webserver, so you can stick with that one. This is where you can set the load balancing algorithm. Your load balancer needs to be able to handle all traffic coming to your site, but it's task is not very computationally expensive, so it will be able to handle much more throughput at a given hardware spec than your webservers. So spec generously, but evaluate your production metrics, so you're not wasting money on a completely overspeced load balancer. You'll also probably want to make your load balancer redundant, so add a second one with floating IP failover (through keepalived for example). This way your load balancer is redundant, but you're also wasting money on a server sitting idle all of the time. If you need to scale your load balancer horizontally, you can always add more pairs and take advantage of DNS round robin to split traffic between them. Alternatively, you can of course buy load balancing as a service through your hyperscaler of choice, or Cloudflare, or whatever. You can also buy load balancer appliances from vendors like Kemp, a lot of enterprise firewall appliances also have load balancer functionality built in. There's also ready-made software solutions available in case you don't want to build the nginx / HAproxy setup yourself and just want a nice setup process with a fancy GUI.

Thank you ❤

Thank you ❤🙏

I got a quite a few people pointing out to me that failures are faster than successes, and I should have pointed that out in the post. I did think about it at the time, but in practice these algorithms already account for that and I didn’t think it helped me achieve the goal of the post 😅 Round robin is a solid choice, and works fine up to much larger scale than most companies will ever achieve.

True but that's kinda not what the article showed. It showed that there are algorithms that are both faster and drop connections less than Round Robin.

I wouldn't know about these posts if not for this channel 🤷🏻‍♂️

I’m glad it shows. My in-progress post is probably the most love I’ve poured into any of them so far. Should be out some time in the next month.

Thanks so much 🙏

Multi-tier load balancing is quite common in practice! The big companies need quite a few levels, at different layers in the stack, to achieve their scale.

A really tricky problem in practice, to the point where I’ve never seen it done. I very nearly didn’t cover weighted round robin because it’s so impractical to rely on humans to judge the cost of anything. Closest I’ve seen is splitting your API endpoints out into groups of “slow”, “medium” and “fast” and treating those buckets differently.

@@samrosewho I was thinking about the API too when i wrote it. But, as you said, "impractical to rely on humans", and we now have something that is sort of good at making predictions, it just needs a lot of data, and well... this is precisely the kind of data we can synthetically generate ;)

The mitochondria is the powerhouse of the cell.

@@samrosewho 🤯

You flatter me! ❤

I talk about this in the post, but be careful assuming your servers are identical! Odds are they aren’t, especially if you’re using VPSs in a cloud provider. Even machines in the same instance class can vary. One of the things I’d love people to take away from this post is that with minimal effort (usually 1-2 lines of nginx config or whatever) you can do quite a bit better than round robin.

@@samrosewho Thanks for the reply. Have you noticed considerable differences in spawning identical VPSs? Depending on your stack, each server can be really simple. Usually, the bottleneck is then the database server, which is a whole different story.

@@rickdg I’ve seen non-trivial differences in servers using exactly the same hardware, it can be pretty wild.

Shared VPS with the same specs can vary a lot. Only if you have dedicated servers you can "bet" that they are equally powerful. But dedicated ones are way more expensive.

The requests are not, even for the same endpoint/entity. Example: list items on account X is light with 0 items. I add 100 items, then it is not. I re-query immediately on the same instance and it is light again, because it is cached.

I have a post about consistent hashing that’s in draft. I haven’t really seen it used in the load balancing space, it’s usually put to work in data sharding from what I’ve seen.

@@samrosewho hey Sam! It was used extensively in a paper about Maglev, Google's distributed load balancer: static.googleusercontent.com/media/research.google.com/en//pubs/archive/44824.pdf

@@samrosewho hey Sam! It was used extensively in Maglev, Google's distributed load balancer. Can't link the paper, unfortunately, since KZfaq deletes comments with URLs for some reason, but it should easy to find. Search query: "A Fast and Reliable Software Network Load Balancer"

I was so annoyed when it was pointed out. I re-read this post at least a dozen times 😅

It’s a static site on GitHub pages. If it ever got overloaded it’d be an excellent day. 😁

Thank you 🙏

Hello night people from the US. You know timezones. People also watch him from Europe 😉

hello night person from the US

I wrote the post and I’m ecstatic Theo used his reach to introduce more people to my work 🙂

Sure, but he still didn't provide anywhere near enough value for this video to be justified. Majority of this video is just reading word for word what you wrote. He could've went out and gathered more blog posts/other materials like yours and just show us parts of those and provide links in description. But that's too much work, that's why this video is what it is

@@VicioGaming and yet you’ve watched it and you are commenting on it

@@nickmoore5105 As if it changes anything. I've watched it to the end to see if Theo actually brings value with this video and, what a suprise, he doesn't

I knowwwww, I have been putting effort into fixing this in subsequent posts. Sorry if this is something that made the post difficult for you. It’s not that I don’t care, it’s that I’m not very good at this yet 🙈

It was actually an awesome explanation, with really nice visuals. I wouldn't know a better way to show the difference without colors either. Amd I'm not a color expert to pick new colors, I just know red/green is a pretty common colorblindness (if thats a word?)

It is! Chrome has tools for emulating different types of colourblindness that I didn’t know about at the time, but I use on all my posts now. I’m starting to branch out into using different shapes and patterns as well, so as not to rely solely on colour 😁