How to use Multithreading with "worker threads" in Node.js?

Рет қаралды 52,327

Күн бұрын

JavaScript is single-threaded, but parallelism and multithreading are possible within Node.js with the help of worker threads. In this video, we're going to explore how one can make an Express.js server faster with worker threads when the main thread is blocked by heavy computations.
📚 Resources:
Stanley Ulili's article on Digital Ocean: www.digitalocean.com/communit...
======⚡⚡⚡======
Consider supporting the channel by clicking the "Thanks" button ❤️
👉 Get a 20% discount on Brilliant.org (to learn Science, Technology, Engineering, and Math): brilliant.sjv.io/kjVO2z
======⚡⚡⚡======
🙌 Become my Patreon and get exclusive perks: / softdevdiaries
💼 Follow me on LinkedIn and drop me a message if you'd like: / gusgadirov
💻 Also, let's connect on GitHub: github.com/gusgad
🕒 Timestamps:
(0:00) Node.js multithreading overview
(3:33) How Node.js worker threads can help
(09:02) Optimizing worker threads
And don't forget to subscribe for more videos like this 😊

Пікірлер: 100

@sohaibnoor6610 Ай бұрын

I have 3.5 years of Experience as a Node js Developer. But I feel I knew nothing. Thank you so much man. This is mind blowing. Love every of it

@guytonedhai Жыл бұрын

This is gold! I have been binge watching your channel. I wish I had discovered your channel much before.

@SoftwareDeveloperDiaries Жыл бұрын

You can’t imagine how much this comment means to me!

@leeshey99 3 ай бұрын

The most intuitive video of multithreading nodejs out there

@vitto_pincharrata Жыл бұрын

An amazing oportunity to use os.cpus()!! + import os from "os"; ... - const THREAD_COUNT = os.cpus().length; and reduce as well! - const total = thread_results.reduce((x, y) => x + y); Thanks for the video! :3

@TheLexa 26 күн бұрын

const { cpus } = require("os"); const THREAD_COUNT = cpus().length; maybe better option

@codingispower1816 6 ай бұрын

Great video! Excited for this whole series!

@jefersoncosta2621 Ай бұрын

This playlist is awesome! Thank you!

@raviagheda5564 3 ай бұрын

Awesome content, Thanks for the creating video Multi Threading. Explanation was too smooth, and especially loved the optimization part.

@SoftwareDeveloperDiaries 3 ай бұрын

Glad you liked it!

@jamshaidsabir5460 4 ай бұрын

You solved a great problem. Thanks a lot

@OleksandrDanylchenko2k Жыл бұрын

Really nice guide! Thanks 🙌🏻

@SoftwareDeveloperDiaries Жыл бұрын

Нема за що, glad you enjoyed it 👍

@Dev-Siri Жыл бұрын

This is the best video I have found that teaches how to use multithreading in Node

@SoftwareDeveloperDiaries Жыл бұрын

Happy to hear that!

@ezetenduwilliams7322 10 ай бұрын

Facts

@arkadiptadas4148 Жыл бұрын

Super helpful. Thank you so much!

@SoftwareDeveloperDiaries Жыл бұрын

Happy to hear, buddy!

@rasulrukh2907 Ай бұрын

great job man!

@ihateorangecat Жыл бұрын

Just finished watching and subscribed. thanks for this amazing tutorial! I actually learnt something really important for my job. So thanks again! and Please upload more tutorial like this. can we also have socket pool tutorials?

@SoftwareDeveloperDiaries Жыл бұрын

Hey, happy to have you on board! And thanks for the feedback, I'll look into socket pools :)

@Infinite_Mainak 11 ай бұрын

Best video I have ever seen on node multithreading👍

@SoftwareDeveloperDiaries 11 ай бұрын

Wow, thanks!

@s.mehradi 2 күн бұрын

Thanks a million! 😊

@vicTor-ft8xv 9 ай бұрын

Thank you brother

@gregoax5474 4 ай бұрын

great content!

@carlosdelgado5632 Жыл бұрын

Excellent explanation, thank you

@SoftwareDeveloperDiaries Жыл бұрын

Glad it was helpful!

@frontend_ko Жыл бұрын

okay, understood, thanks for nice tip

@acluster3411 Жыл бұрын

Very well done!

@SoftwareDeveloperDiaries Жыл бұрын

Thank you!

@ahmedsoran4710 Жыл бұрын

thank you for sharing this was so helpful

@SoftwareDeveloperDiaries Жыл бұрын

Sure thing 🙂

@angryman9333 Жыл бұрын

bzhy :d

@pabloroque8268 3 ай бұрын

Great content, i hope you to do more videos with advanced concepts and proyects with nodejs, best regards from Argentina

@SoftwareDeveloperDiaries 3 ай бұрын

Gracias amigo! More to come 😉

@Abdulkalam-he7wo Жыл бұрын

great work , explanation at its best..!

@SoftwareDeveloperDiaries Жыл бұрын

Thanks mate!

@Abdulkalam-he7wo Жыл бұрын

Please do more stuffs like this💥

@alirezamosavi6185 10 ай бұрын

Great bro

@Jberv Жыл бұрын

Deym, I thought i'm on my limits for optimizing my api, Thank you so much for this

@FreeDiver-by7fz 20 күн бұрын

Google is playing a joke on me. I spent a month fiddling with the first topic, with the promise of all cycles. I just solved the problem and KZfaq immediately suggested your video...

@verdynandreyhans9592 Жыл бұрын

thanks bro,

@parassharma7041 2 ай бұрын

Great video

@SoftwareDeveloperDiaries 2 ай бұрын

Thanks!

@caotichcuc 2 ай бұрын

God, it's super helpful. Please make more videos about APIs optimization and security for Nodejs, pleaseeeeeeeeeeeeeeeeeeeeeeeeeee

@labidyramzi4840 8 ай бұрын

Question: What if I will be using "docker + kubernetes" to horizontally scale my node app and at the same time i have a file upload + file processing intensive task in my app should I still use "worker threads" to manage the intensive tasks or "docker+kubernetes" will solve the problem anyway ? Thank you for this amazing explanation !

@Aleks-fp1kq 4 ай бұрын

Hello. why do you need the number of cores? If I understand it correctly, the new loop is located in the same process but on a different thread.

@Pareshbpatel 6 ай бұрын

An excellent tutorial on Multithreading in Node using Worker Threads. Thank you very much. {2024-01-12}, {2024-01-15}, {2024-01-19}

@muharief3885 Жыл бұрын

great for single good server, bad for container. This way was just make container suffer bcose the resources already taken up by the io operation.

@muhammedhuzaifa1392 19 күн бұрын

I have a question, If I cluster node js app on 8 cors and if I use thread on each computation in my code, So assumption is my same code is running on 8 clusters and all have threads in them mean to say the api I wrote in my single code is running on 8 different cors, does it improve performance and if it improve performance? Can you tell me pros and cons of it?

@vaibhavm1986 Жыл бұрын

Hi , one question is like I have scenario in which have to do file operation many times we are using some third party library to club or combine small doc files to one big final pdf , but that has to be in particular order , there are 20 sections , each section is tree containing further sections and so on and so forth ,each section contains docx file that has to be merged, so can I divide workers to club 20/ thread-count and then finally club the result of all of them to final file, only thing is I would have to find thread count , programmatically do you agree? is there any other way you can recommend

@SoftwareDeveloperDiaries Жыл бұрын

Maybe try worker threads? I have a video on it too. If that doesn't help, you can try asking ChatGPT, I'm pretty sure it knows everything :d

@vaibhavm1986 Жыл бұрын

@@SoftwareDeveloperDiaries thanks

@firoj.siddiki 9 ай бұрын

subscribed.

@SoftwareDeveloperDiaries 9 ай бұрын

Happy to have you here, mate!

@ayeshazahid2421 Жыл бұрын

awesome video but I have a question, can you elaborate on the difference between background jobs and worker threads. I have a situation in which I need to call db recursively then call third-party API which takes some time to respond then refactor data then filter specific data and again call third-party API and then bulk insert into db + I need to perform this action inside cron jobs which get triggered after every 3 hours so what should I need to use???? and I'm using ec2 which has only 1 core so I think I can't divide it if my solution is worker thread

@SoftwareDeveloperDiaries Жыл бұрын

Thank you! :) Sounds like you need a queueing mechanism for such a bulky operation. Take a look at RabbitMQ or Apache Kafka and how you can integrate them with your existing API.

@ayeshazahid2421 Жыл бұрын

@@SoftwareDeveloperDiaries okay 🤩 thanks again sir

@ronnykgalema 10 ай бұрын

Imagine combining multithreading with clustering? That would be a bomb 💣

@SoftwareDeveloperDiaries 10 ай бұрын

Hmmm interesting!!

@jsarabia_dev7371 3 ай бұрын

If our app is inside of pods of kubernetes is not possible the Multithreading if has not have more 1 CPU for pod?

@SoftwareDeveloperDiaries 2 ай бұрын

That's a great question! In a Kubernetes environment, each pod typically runs a single container, and the number of CPUs available to a pod depends on the resources allocated to it. If a pod has only one CPU allocated, then indeed, multithreading might not provide significant benefits since there's only one CPU core available for processing.

@Black-xq9ey 2 ай бұрын

Sir Which theme you are using? It's cool for the eyes.

@SoftwareDeveloperDiaries 2 ай бұрын

Hey mate, it's called "Palenight (Mild Contrast)"

@ahmeterdonmez9195 5 ай бұрын

hello brother. Can we use this to handle 2-3 different tasks at the same time? I subscribed your channel ;)

@SoftwareDeveloperDiaries 5 ай бұрын

Yes, you totally can :) Thanks for the subscription!

@vb4190 Жыл бұрын

and what will happen when you run out of all your worker threads? in case of multiple requests to blocking endpoint by different clients

@SoftwareDeveloperDiaries Жыл бұрын

The machine will have to launch a fair amount of worker threads and get noticeably slow, before it throws an error that it can't launch a new thread. But this is probably the sequence that you'd see.

@vb4190 Жыл бұрын

looks like a case that should be handled)

@vaibhavm1986 Жыл бұрын

Hi, I did not understand the idea of 7 threads , kindly correct me If this is wrong from what I know is UV_THREADPOOL_SIZE = 4 , by default and these are the only threads used by libuv for some of the operations! i.e except CPU intensive tasks

@SoftwareDeveloperDiaries Жыл бұрын

Correct. When we refer to a "thread", we actually mean a "thread pool". With UV_THREADPOOL_SIZE you define the size of the pool, which by default consists of 4 "child threads". You can still increase the size of the pool up to 1024.

@vaibhavm1986 Жыл бұрын

@@SoftwareDeveloperDiaries is it practically possible to have 1024 threads ?

@vaibhavm1986 Жыл бұрын

my cloud setup is having 7 nodes each node has 2 vcpu , virtual CPU I am not sure how many threads each instance has , I printed cpus length using os module it printed 2 , so only 2 threads ? do you suggest any other way or to know about it

@vinaykumarkharwar177 10 ай бұрын

@@vaibhavm1986 It can't, the UV_THREADPOOL_SIZE can't be bigger than the number of total logical threads on your CPU. If you have an Octa-Core CPU each of them have 2 threads, then the logical threads at your disposal will be 8 * 2. It can't be more. To automate the code, you can use - const os = require("os") process.env.UV_THREADPOOL_SIZE = os.cpus().length Also, the threads used in the worker pool are not real threads. They are circumventive solution for multithreading. Possibly very close to the Co-routines of Kotlin language. Or GoRoutines from Go lang. In kotlin, you can have thousands of coroutines with multiple tasks. If you are looking for multithreading, you can use cluster module from node.js, which spawns many worker instances on each cpu thread, but yeah they are still memory intensive and mixing worker_thread with clustering is bad idea.

@atmai1026 3 ай бұрын

I has a question that when we use Multithreading with "worker threads" or real applications of it. I have saw many Nodes courses and they just introduce about it, not talk about the real application

@user-gy9sv7fp3i 3 ай бұрын

Technically, node does not use threads for network operations. It uses different mechanism called async io. But the idea is similar - main thread is also not blocked.

@user-nw3rz4yp4u 5 ай бұрын

I think while creating worker threads , new main thread is not created. Correct me if I am wrong.

Ай бұрын

0:19 - wait, I'm not ready yet!

@SoftwareDeveloperDiaries Ай бұрын

Ooops 😁

@reactjs1900 11 ай бұрын

What is the difference between workers and fork or child process

@jperusm 4 ай бұрын

Processes take much more resources to spawn while threads are lighter and are handled by the same process

@chavdarajan7428 3 ай бұрын

ACCHA KAM 👍 - HANCOCK

@esp2644 Жыл бұрын

How download code example?

@SoftwareDeveloperDiaries Жыл бұрын

Hey, there's none. I was hoping you could reproduce it by watching the video

@swagz6440 Жыл бұрын

How hard is it to write that

@aki1840 Жыл бұрын

Thanks for this video! All Node haters - in your face! :D

@SoftwareDeveloperDiaries Жыл бұрын

Hell yeah hahaha

@FreeDiver-by7fz 20 күн бұрын

Just learn my friends, and you will get where you want

@meandradethiago Жыл бұрын

Jesus loves you, man. Thanks for this incredible video.

@SoftwareDeveloperDiaries Жыл бұрын

My pleasure! 😊

@innoventstack Жыл бұрын

Hey Use cluster module instead of this much better approach

@SoftwareDeveloperDiaries Жыл бұрын

I will check it, thanks

@Flonfl0n 3 ай бұрын

Great video, but you should have used a reduce to sum your results. I'm triggered haha

@SoftwareDeveloperDiaries 3 ай бұрын

Check this out, it's better to use good old for loops when working with a lot of data :) leanylabs.com/blog/js-forEach-map-reduce-vs-for-for_of/

@Flonfl0n 3 ай бұрын

@SoftwareDeveloperDiaries I meant to sum the result of each thread. Total = thread_results[0] + thread_results[1]....

@SoftwareDeveloperDiaries 3 ай бұрын

@@Flonfl0n oh i see ^^

@Flonfl0n 3 ай бұрын

@@SoftwareDeveloperDiaries hehehe