Making multiple HTTP requests using Python (synchronous, multiprocessing, multithreading, asyncio)

Рет қаралды 88,046

4 жыл бұрын

In this video, we try to explore various ways using which we can execute multiple HTTP requests using Python. We try synchronous and asynchronous techniques and find out which one works best.
Code: github.com/nikhilkumarsingh/a...
Multiprocessing in Python: • What is Parallel Progr...
Multithreading in Python: • What is Multithreading...
Concurrent Programming in Python (asyncio): • What is Concurrent Pro...
------------------------------------------------------------
Explore my tutorials: nikhilkumarsingh.github.io/tu...
More awesome topics covered here:
WhatsApp Bot using Twilio and Python : tinyurl.com/y9n9ott5
Serverless Rest API using AWS and Python : tinyurl.com/y8yg4vem
Creating Chat Application using Flask, Socket.IO & mongoDB : tinyurl.com/y8kbn4xu
Curses in Python : tinyurl.com/y8ws3x4l
Discovering Hidden APIs : tinyurl.com/y7sdegoq
RegEx in Python : tinyurl.com/y84cnuns
Python for Data Science : tinyurl.com/yd7x6j5e
Introduction to Pandas : tinyurl.com/ycr4qytt
Introduction to Matplotlib : tinyurl.com/y8caem3r
Introduction to Numpy : tinyurl.com/yc8uhtn9
Functional Programming in Python : tinyurl.com/ybubzum6
Python Package Publishing : tinyurl.com/ya2fn3wb
Multithreading in Python : tinyurl.com/ycnummb8
Multiprocessing in Python : tinyurl.com/y8qkjqtx
Parallel Programming in Python : tinyurl.com/yakl5udt
Concurrent Programming in Python : tinyurl.com/y9ppkkhc
Dataclasses in Python : tinyurl.com/y8zfqleq
Exploring KZfaq Data API : tinyurl.com/y7jhs8rm
Just For Fun : tinyurl.com/y9on8gb5
Exploring AWS : tinyurl.com/ydxmhhz9
Jupyter Notebook (Tips, Tricks and Hacks) : tinyurl.com/y8qdhcrp
Decorators in Python : tinyurl.com/y7ghraa9
Inside Python : tinyurl.com/yd8xqbjj
Exploring datetime : tinyurl.com/y96jzylm
Collections in Python : tinyurl.com/ybxltgvk
Networking : tinyurl.com/y97eroql
Computer Vision for noobs : tinyurl.com/y8pxz436
Python for web : tinyurl.com/y829py3g
Awesome Linux Terminal : tinyurl.com/y8no3zso
Intermediate Python : tinyurl.com/yasmecaw
Tips, tricks, hacks and APIs : tinyurl.com/y9xqzzl4
Optical Character Recognition : tinyurl.com/ybt3to6q
Facebook Messenger Bot Tutorial : tinyurl.com/y9wvjc9g
Facebook: / indianpythonista
Github: www.github.com/nikhilkumarsingh
Twitter: / nikhilksingh97
------------------------------------------------------------
#python #asyncio

Пікірлер: 86

@IndianPythonista 4 жыл бұрын

Code: github.com/nikhilkumarsingh/async-http-requests-tut Multiprocessing in Python: kzfaq.info/get/bejne/gNtkq86nq9OUZ6M.html Multithreading in Python: kzfaq.info/get/bejne/kLZ9a7d035vHlGw.html Concurrent Programming in Python (asyncio): kzfaq.info/get/bejne/r55lepp4sb_MiXU.html

@ibrahimaba8966 3 жыл бұрын

.. can we use multiprocessing and multithreading at the same time ? I mean having n processes and m thread per process.

@savantofillusions 2 жыл бұрын

Your snake art got me to stay

@Millez Жыл бұрын

0:59 Synchronous 4:23 Multiprocessing 7:43 Multithreading 15:23 AsyncIO

@praisedavid1653 3 жыл бұрын

This is the most informative video I have seen on these features/modules. Kudos

@learnerb3403 Жыл бұрын

Incredible video sir with clarity in presentation and in depth knowledge. Thanks for sharing your knowledge. Looking forward for more such real time example videos.

@samirbhattarai3441 3 жыл бұрын

Nicely demonstrated. Great.👏

@bufmouse13 2 жыл бұрын

Incredible video, thank you! Im trying to find ways to turn three scripts that run a AI art generating bot into one and I think some of your examples is really going to help.

@eduardogil1775 3 жыл бұрын

Excellent explanation, Thanks for sharing! 👍🏼

@zameerahmed1775 2 жыл бұрын

Ur too good .............. youthful & motivating voice. Above all ur python concepts are up to the mark.

@mashoodurrehman9840 4 жыл бұрын

Awesome.... keep making stuff like that.

@kiyotaka627 3 жыл бұрын

thank you for this well designed video, very informative.

@poojagautam1406 3 жыл бұрын

Amazing tutorial! Thanks a lot.

@EnglishRain 3 жыл бұрын

Brilliant video, thanks a lot!

@AdityaShinde-cr5iv Ай бұрын

Thank you for this video. Nice explanation.

@SurendraTarai 3 жыл бұрын

you are awesome, well explained... appreciate it. keep it up

@krishjodu3767 2 жыл бұрын

Thanks for your wonderful video bro, It's raising AssertionError at timer

@_sharmaji.dev_ 3 жыл бұрын

AWESOME!!!! REALLY HELPED.

@adnankaya 2 жыл бұрын

Great tutorial! Thanks for sharing.

@unemploymentisacurse 4 жыл бұрын

You explain pretty well sir.... Learning Python from you is such a honour for me.

@unemploymentisacurse 4 жыл бұрын

God save my english.

@zakaria.morchid 4 жыл бұрын

I like those stuff here too

@vishalwaghmare3130 2 жыл бұрын

Best video on this topic

@chaarlog5508 3 жыл бұрын

Very informative video, nice work bro

@shokhrukhabduahadov3985 3 жыл бұрын

would be better if u put bookmarks to the video parts

@dm-vh9yu 4 жыл бұрын

Great video, keep it up

@nelsonkenmochi9353 3 жыл бұрын

Nice video, thanks for that. :)

@lazypunk794 3 жыл бұрын

Great job mate

@rajii515 9 ай бұрын

Thankyou for this video by the way.. it helps me a lot 👍👍

@techqsr Жыл бұрын

Informative 😍

@gregorybutcher2647 3 жыл бұрын

Awesome video! Just one question: Do take different amount of processing power? if so which is more efficient, multithreading or asynio?

@sulaymanali8080 3 жыл бұрын

really great. thanks a lot

@drissaitkassimusic 3 жыл бұрын

Best video to understand this tipic

@arjunchatterjee5667 2 жыл бұрын

superb video!

@veereshgs5298 2 жыл бұрын

Great 👍

@AngelGonzalez-kl8oi Жыл бұрын

Great!

@googlemonkey7746 3 жыл бұрын

Nice explained

@m.alaiady3627 2 жыл бұрын

Nice explanation

@TanviAravind 4 жыл бұрын

Thanks for making this video of Asyncio hope it covers web scrapping

@TanviAravind 4 жыл бұрын

I had finished python asyn await and aiohttp during the weekend... earlier it used to take 40 to 50 mins to scrap the data after aiohttp and Asyncio implementation it's just 90 Seconds

@unemploymentisacurse 4 жыл бұрын

@@TanviAravind yea it's such a time saving during projects.

@michelromero7671 2 жыл бұрын

Thanks for this video, I didn't actually watched it because of the great thumbnail that told me what I came looking for. (I left a like tho)

@sudhakarpandey693 2 жыл бұрын

If someone explaining so nicely then plz don't mark 👎...if he is missing something then tell him personally.... don't make his moral down...I found his vedio perfect 👍thanks frnd

@hgt3 9 ай бұрын

thank you

@deepanshuchoudhary4598 2 жыл бұрын

God bless you

@user-hk4bq8be1d 2 жыл бұрын

этому типу надо памятник поставить( в хорошем смысле :) )

@getconsiousstate 3 жыл бұрын

Any reason why ProcessPoolExecuter is not used for multiprocessing ? and what about multiprocessing.pool.ThreadPool ? Any Idea why these many modules are created for same functionality in Python ?

@Berkeozbeek 2 жыл бұрын

Could you please tell me how I can return the data printed from fetch() function in multithreadding? Whenever I tried to return it, t always gives me None for some reason but I can print it without any problem.

@mahdisecurity1700 3 жыл бұрын

hi your video is very good subject. but for later video please : multiple function multiple sync multiple function multiple async multiple function multiple threading multiple function multiple processing and combine all concepts with simple language . thank you very much . god willing

@vsanjairaj 3 жыл бұрын

I have to get status codes for 150k urls in a data frame. I want a new columns 'status' in the df and i want the response code of the request of url. What's the best way to do it?

@JamalanJuda 2 жыл бұрын

What if I have a list of different URLs that I want to do GET from? How can I apply multithreading in that scenario?

@maba7097 Жыл бұрын

Thank you for the video I have an app that I integrated with api. I send a message to it and it analyzes it and then returns a result t, but when I send many messages at the same time, it does not go well and you get a mess In my case, what solves my problem from what you mentioned?

@TheJBardopoulos 3 жыл бұрын

and what's the limit for max-workers (depends on my PC, I know, but how to find the limit?)

@yellie5629 3 жыл бұрын

Can I organize and post your lecture on my blog, Medium? I will leave a reference.

@mithunnambiar1433 3 жыл бұрын

An assertion error is stopping me from the synchronous method. does anyone experienced the same problem?

@trickdigger5744 3 жыл бұрын

Sir, Kindly check the pull request made on github.

@bhaveshmevada8424 3 жыл бұрын

Bro i want to post json data with requests.post method and i did it but it takes very long time to send all data...plz help me

@akhil3365 3 жыл бұрын

what is the difference between qrequests and asyncio

@YuvrajSingh-pz7wt 4 жыл бұрын

Can u plzzz makr video on topic of what is channel in django and make chatting app ..plz sir

@PGhai 3 жыл бұрын

Nikhil Bhai, do u also teach one on one?

@IndianPythonista 3 жыл бұрын

Hi. No plans to do that yet.

@ncjatin 2 жыл бұрын

I have a question... Example in synchronous code.... We are only defining methods not calling it... Than also methods are running

@KirillBezzubkine 2 жыл бұрын

The video is good. BUT: how would you make 10000 http requests when the api supports about 50 requests only at a time (otherwise it throws 429 error)? I need help with that. Thank you

@kattynip0 2 жыл бұрын

need to rotate IP

@saurabhverma6108 Жыл бұрын

@@kattynip0 what solution did you find for it?

@bhola8023 Жыл бұрын

Which OS u are using and what's that theam

@peterchao7429 2 жыл бұрын

Great video, I have learned a lot here. One question I have is, instead of get HTTP request, what if I want to do post request?

@giorgosK936 Жыл бұрын

naughty boy

@abhisekkeshari4850 Жыл бұрын

Could you please add the timestamp of the methods that would be a great help

@yt-1161 2 жыл бұрын

@13:46 what are the downsides of having too many max_workers ?

@ashishsondagar2812 2 жыл бұрын

15:25 asyncio

@Duychienvt Жыл бұрын

I think threadings are running inside one process. It is impossible to do faster than multil-processing. The reasons that your multithreading is faster : may be this is running multithreading inside multiprocessing Check out this video: kzfaq.info/get/bejne/m8l7isySrampe6c.html

@kirtimali723 2 жыл бұрын

my main function is not calling... can you help me with it?

@ncjatin 2 жыл бұрын

Just cross-check.. have you out @timer(1,1) before main function ?

@radiozamindar4710 3 жыл бұрын

hello bro can i have some help ?

@FRUXT Жыл бұрын

Why no subtitles ?

@sahindian 4 жыл бұрын

Sir, can you let me know how to integrate asyncio to call multiple cameras to take a multiple angle picture once at a time.like using picamera over raspberry Pi. Thank you

@MrPedala94 3 жыл бұрын

sounds to me like you can just do that in a sequential manner.

@nirakarpadhy3116 3 жыл бұрын

!!! PETITION for making tutorials using examples of "POST" requests !!! @IndianPythonista Why does every tutorial and blog post always assume we only only need "get" requests? I try to implement the same code for "post" requests and I run into all sorts of errors. I wish you or someone on this earth for once did tutorials for fast http post requests.

@switchblade3868 3 жыл бұрын

Ahats that linux???

@kumarnitish6552 3 жыл бұрын

Are you sure of the time calculation using a timer, because it seems like coding is taking more than what it prints in the console? Your timer logic is not correct