Why thread pools even exist? and how to implement them?

Рет қаралды 35,884

Күн бұрын

System Design for SDE-2 and above: arpitbhayani.m...
System Design for Beginners: arpitbhayani.m...
Redis Internals: arpitbhayani.m...
Build Your Own Redis / DNS / BitTorrent / SQLite - with CodeCrafters.
Sign up and get 40% off - app.codecrafte...
In the video, I explained the importance of thread pools in managing multiple concurrent requests efficiently without overwhelming the system. I discussed the drawbacks of spinning up new threads for each request and how thread pools help mitigate these issues by capping the maximum number of threads. I highlighted real-world use cases and provided a quick prototype implementation in Go. Tuning thread pool size based on hardware and workload characteristics was emphasized, along with practical tips for implementation. Watch the video for a deeper understanding of thread pools and their implementation.
Recommended videos and playlists
If you liked this video, you will find the following videos and playlists helpful
System Design: • PostgreSQL connection ...
Designing Microservices: • Advantages of adopting...
Database Engineering: • How nested loop, hash,...
Concurrency In-depth: • How to write efficient...
Research paper dissections: • The Google File System...
Outage Dissections: • Dissecting GitHub Outa...
Hash Table Internals: • Internal Structure of ...
Bittorrent Internals: • Introduction to BitTor...
Things you will find amusing
Knowledge Base: arpitbhayani.m...
Bookshelf: arpitbhayani.m...
Papershelf: arpitbhayani.m...
Other socials
I keep writing and sharing my practical experience and learnings every day, so if you resonate then follow along. I keep it no fluff.
LinkedIn: / arpitbhayani
Twitter: / arpit_bhayani
Weekly Newsletter: arpit.substack...
Thank you for watching and supporting! it means a ton.
I am on a mission to bring out the best engineering stories from around the world and make you all fall in
love with engineering. If you resonate with this then follow along, I always keep it no-fluff.

Пікірлер: 77

@LeoLeo-nx5gi 6 ай бұрын

A very IMP question I had. 1. So we know that usually when the run() method of a thread is executed, the thread dies automatically after it's work is done. 2. Then why is it said that we put the same thread back to the pool or how does ThreadPool does that (i.e reuse the same thread), I think we are again creating a new thread right? and putting that back to the pool? and not reusing the original thread (which did the earlier task). Would be great if you could guide here. Thanks for the knowledge share !!

@AjayKumar2 6 ай бұрын

You are correct in saying that a thread dies after returning from the run() method. The key here is to never return from the run() method. This is an implementation detail of the ThreadPool class (in Java context). A common pattern is to maintain an array of threads (worker threads) and read tasks from a queue. When a new task arrives, a thread from the pool is assigned to it if it's available else the task waits. As soon as the thread is done executing the task, it goes back to the queue to get a new task else waits for it to arrive. This way, the lifecycle of the worker threads are maintained by the owning ThreadPool object which is executed in the main thread. Worker threads are alive as long as the ThreadPool object is alive.

@reudrax 6 ай бұрын

This concept is same as connection pooling. saving the time to make connections. putting a limit on numbers of concurrent db connections that can be made, thus handling the load effectively

@jakubfrei3757 3 күн бұрын

Best explanation i came across, thank you

@shawkishor 6 ай бұрын

In one of the faangs, there was an interview question: one method (class method/REST call etc) is being called by 1 million users, all users get the response at no delay. What is happening behind the scene?how come that method is getting executed million times concurrently.

@SampathKumarKamati 6 ай бұрын

There must be more than one server handling the requests that are distributed by a load balancer

@shawkishor 6 ай бұрын

@@SampathKumarKamati given the same answer even threadpools on web server too. Interviewer not satisfied

@freddyv5353 6 ай бұрын

response at no delay? that’s impossible. did you ask if the response was valid? they may all be getting errors

@shawkishor 6 ай бұрын

@@freddyv5353 how? Suppose thousands of users are ordering different products at the same time. I have an order create API that is handling the requests concurrently.

@sasha751 6 ай бұрын

Resource is cached 😂

@burionyt 6 ай бұрын

Hey awesome video but here is a tip on "thread pool in Golang". In go thread pool is an anti-pattern and you should generally use semaphores instead. One can implement a basic semaphore using a buffered channel. There is a weighted semaphore implementation in the golang/x pkg if you wanna checkout or could make one yourself(I generally do this)

@jspnser 4 ай бұрын

For the topics in future: I wish you could explain internals of go routines more in depth, (why they're light weight etc), thank you for the video

@keerthi1070 6 ай бұрын

Another main reason for thread pool is "creating new thread and deleting then itself is very costly in terms of CPU". Also what is the max thread for a n-core cpu ?

@hiteshbitscs 6 ай бұрын

In my experience in special VM means your workload is running.. I would say 2 * n should be size of thread pool... the reason is each core support 2 thread due to hyper threading concepts in modern chips.... so ideally across your application number of threads should not exceed 4 * n cores... take with a pinch of salt!!!

@durgaprasadreddy5033 6 ай бұрын

What is costly?

@_sudipidus_ 6 ай бұрын

just a nitpick: you demonstrated with an example of go routines which are not actually threads (even using just go routines without keeping a worker pool would perform better than threads) but on a positive side, this was more about generic pooling concept (pooling of threads, HTTP connections, db connections or anything whose creation/bookkeeping is expensive)

@AsliEngineering 6 ай бұрын

Yes. I tried to keep it slightly simple. If I would have covered the specifics, it would have repelled a lot of folks 😅

@hiteshbitscs 6 ай бұрын

You mentioned for network bound workload we can have more threads, for CPU heavy workload we have few threads... Isn't it should be opposite? 1. For network I/O workload because you wait more on network packet to send/arrive due to it's millisecond latency.. we can have few threads. so that thread has works rather than doing short stuff and wait 2. For CPU heavy workload : because you have more works we have more threads.. (Here I guess in hyper threading each core has 2 threads per core.. hence max we can go is approx/. 2 *number of cores) at max...

@Ðogecoin 4 ай бұрын

Thank you for this insightful explanation Mr armpit

@user-qu5zp8pl8l Ай бұрын

Hi Arpit thank you for the knowledge, just one question here. The implementation mentioned in the video is through go routines. In this case what does the thread pool signifies in terms of operating system threads, more briefly If i create a thread pool of size 5 (similar to mentioned in the video) will it reserve 5 go routines or 5 platform threads ? As afaik we can have more than one go routine running on one platform thread, due to the fact that go scheduler itself takes care of that. And a follow up on this is, If I have to implement this In java do I use the normal thread pool or virtual threads ? Thank you, cheers

@AsliEngineering Ай бұрын

Thread pool is all about keeping a set of threads ready to be pulled out and use for whatever you need. Virtual threads in Java were inspired from Goroutines. The concept of pools is independent of threads or virtual threads. The idea is to optimize on the cost of creation of new threads by reusing the existing ones.

@durgeshkhot7811 5 ай бұрын

underastood, by your video

@sachinmalik5837 3 ай бұрын

Hey Arpit, I know it a old video, but if you do see this and have a seconds to spare, I have a doubt. Why do we have a unbuffered channel, here like if we are adding jobs, to a workQueue, Should our channel have some buffer, in above code/scenario, are we blocking the putting of the job on workQueue, because it is unBuffered, so I add a job, I have to wait for some thread/go routine to pick up that job? before adding new Jobs to queue I am new to Go, and I am loving what you have to teach, Thanks

@AsliEngineering 3 ай бұрын

You can use an buffered channel here, but that would not change the concept or the approach. That still remains the same. I just kept things simpler for the demo :)

@sachinmalik5837 3 ай бұрын

@@AsliEngineering Awesome. Thanks for reverting back.

@FaintArt 6 ай бұрын

that's a damn good handwriting

@joelphilip2942 6 ай бұрын

if thread pool is 5 and no of tasks > 5, doesn't it execute parallelly (not concurrently) ?

@t1m3__ 6 ай бұрын

great video, thanks!

@sharoonaustin551 6 ай бұрын

Isn't it something similar to the batch processing (Correct me If I am wrong) For example: let's say you have a slice which contains 40 elements and at a time you want to process 20 of them, so you'll fetch the first 20, iterate over them and keep adding to the wait group, and perform the task using go routines and outside the loop you added wg.wait(). This way you're creating 20 threads and running them for a batch isn't it?

@AsliEngineering 6 ай бұрын

You are assuming all tasks will take equal time to complete. Which might not be the case.

@AsliEngineering 6 ай бұрын

What all topics do you want me to cover? Do add them as a reply to this comment ⚡

@PrashantKumar-zz5zc 6 ай бұрын

Oauth

@codingvibes3537 6 ай бұрын

Great explanation sir!! Please make the next video on internals of redis database. Very much curious in deeping down on this topic with you.

@arpanmukherjee4625 6 ай бұрын

Implementing a SSTable based data storage layer from scratch.

@dharsanr6504 6 ай бұрын

Sir if you can please make a video series on building real word projects from scratch ( like the one's which are there in BUILD YOUR OWN X repo) Eg : * creating our own compiler * Git * HTTP server * Network application without libraries

@_ipublic 6 ай бұрын

Can you bring more such contents on threads considering more bottleneck problems. thanks in advance!

@malaypatel4014 5 ай бұрын

One of the best content 👌

@AsliEngineering 5 ай бұрын

Thank you Malay!

@user-yf1ge4mj4m 6 ай бұрын

Your videos are Netflix series to me❤

@BhanuReddyputtareddy 6 ай бұрын

What is the font you have used in vscode looks nice

@neo_otaku_gamer 6 ай бұрын

Thank you for explaining the concept on thread pool. How can we implement similar concept in python, such as server less functions on cloud. Python inherently doesn't have support for threads and thread pool due to Gil.

@da5m0n 6 ай бұрын

For our server example there is also another way to handle client, epoll/kqueue, so my question is what is the tradeoff of using thread pool over epoll or vice versa? Specially for the I/O bound task.

@mayankujawane3147 6 ай бұрын

We have so many server side languages, so how to decide which language to choose. What are the factors which can help to make the decision. Mainly for Golang, Java, Javascript, Python, C#

@RahulSsup 6 ай бұрын

It’s a decision you make based on many factors : comfort-level of Developers, performance expectations, task domain etc. Personally, i think go gives a good balance between the two. I would choose Python, if the work is more data science focused. Javascript for Server-Side only if the developers are JavaScript lovers and cant touch anything else.

@MarathiNationOne 6 ай бұрын

If cpu cores are 4 and threads are 10 is it worth? Or can we have at most treads equals to no of cores

@undergraduate6050 6 ай бұрын

Programming language agnostic Concurrency control mechanism.

@SurajYadav-yd2cu 6 ай бұрын

Great explanation ! Just wanted to ask , will the worker threads created in the pool keeps on looking for a task submitted ? Or they only get active when a task is submmited ?

@AsliEngineering 6 ай бұрын

they get active when there is work to do, otherwise they sleep. It is wither managed by OS if kernel thread, and by language runtime if user threads.

@Chakree45 6 ай бұрын

Aa golang is your fav language, why don't you upload golang related videos?

@AsliEngineering 6 ай бұрын

If by Golang videos you mean language tutorials, then I am not keen on doing it because I believe language learning can happen the quickest through books and blogs. Hence I do use Go in most of my examples to explain a concept or showcase how some problems can be solved.

@Chakree45 6 ай бұрын

@@AsliEngineering thanks for the reply, could you please suggest which books or blogs to follow to learn golang better and fast. Thanks in advance:)

@AsliEngineering 6 ай бұрын

@@Chakree45 I have the books I referred to for Go on my bookshelf ArpitBhayani.me/bookshelf

@Chakree45 6 ай бұрын

Thanks arpit and all d best to u🫂

@AsliEngineering 6 ай бұрын

@@Chakree45 thank you 🙌