How to Make 2500 HTTP Requests in 2 Seconds with Async & Await

No video

How to Make 2500 HTTP Requests in 2 Seconds with Async & Await

Рет қаралды 114,923

2 жыл бұрын

DISCORD (NEW): / discord
This is a comparison about how to use Async and Asynio with AIOHttp and Python vs using threads and concurrent futures to best understand how we could make several thousand http requests in just a few seconds. Learning how to do this and understanding how it works will help you when it comes to running your own servers and web services, and stress testing any API environments you offer.
github.com/jhn...
text articles from: realpython.com/
Support Me:
Patreon: / johnwatsonrooney (NEW)
Scraper API: www.scrapingbe...
Proxies: proxyscrape.co...
Amazon UK: amzn.to/2OYuMwo
Hosting: Digital Ocean: m.do.co/c/c7c9...
Gear Used: jhnwr.com/gear/ (NEW)
-------------------------------------
Disclaimer: These are affiliate links and as an Amazon Associate I earn from qualifying purchases
-------------------------------------

Пікірлер: 107

@jmoz Жыл бұрын

You do not need to explicitly create task on line 18

@dpm_07 6 ай бұрын

why so ?

@PixelThorn 4 ай бұрын

Gather already accepts coroutines, so any async function can be supplied directly

@victorhaynes508 9 ай бұрын

I can't believe how fast you got to the point. thank you for reading the room

@swordlion294 Жыл бұрын

ThreadpoolExecutor is slower than Thread but saves memory. You don't need to use asyncio for waiting. You can use joins, barriers, wait groups and mutex locks with Thread to achieve the same. It comes down to preference, although asyncio is more streamlined, and thread is better suited for manual optimization that requires utmost speed and care.

@JohnWatsonRooney Жыл бұрын

Thanks for the info. I still have a lot to learn about threads!

@ErikS- Жыл бұрын

"how to build a ddos attacker..."

@dpm_07 6 ай бұрын

Right 😂😂

@amosrocha6793 Ай бұрын

Kkkkk

@ihateorangecat 2 жыл бұрын

hey sir👋 as a self taught dev You are one of my inspiration 🙌🙌🙌 i started learning web scraping recently. i downloaded some videos about scrapy and BeautifulSoup tutorials of yours and i followed along them and found pretty comprehensive and clear i did learn better by your videos. i do hope more and more tutorial videos!!! thanks million times !🙏

@JohnWatsonRooney 2 жыл бұрын

Thank you that is very kind!

@RickeDz 9 ай бұрын

I am scraping a lot of things on web, this is just a pretty thing to do, I like this too much, it's so beautiful to see all the thing beeing "gathered" ahaha, are you still on this thing?

@fmanca100 3 ай бұрын

John, your channel is absolutely fantastic! congratulations!

@JohnWatsonRooney 3 ай бұрын

Thank you very kind

@fmanca100 3 ай бұрын

@@JohnWatsonRooney I don't say this lightly, best educational content I have seen in years. Straight to the point, rich and very well explained! (ok, and I now I stop :-)

@bakasenpaidesu Жыл бұрын

u can speed up threading by using maxworkers = x with concurrent.futures.ThreadPoolExecutor(max_workers=10_000) as executor: executor.map(scraper_sub, links) max workers = 10_000 means 10k workers work at same time OR a single worker work 10k work at a time

@k98killer 13 күн бұрын

Threads do not actually compute simultaneously in CPython because of the GIL. The core dev team is experimenting with removing the GIL, but it will be years before it becomes a production-ready option. Iirc, micropython does not have a GIL, so it is able to actually execute threads simultaneously.

@JohnWatsonRooney 13 күн бұрын

thanks for the clarification

@bobong4248 2 жыл бұрын

Hi good sir thanks for the vid and most importantly, being actively engaging with people in the comment section Cheers

@JohnWatsonRooney 2 жыл бұрын

Thanks I’m glad you enjoyed the vid 👍

@artemfagradyan3890 Жыл бұрын

Thanks for your video!! Tried to send multiple async request for some api but my previous code was unefficient and your example helped me!!

@JohnWatsonRooney Жыл бұрын

Great stuff glad it was useful for you thanks!

@aprioriprogrammer9100 2 жыл бұрын

THIS IS FREAKING AWESOME, THANKS MAN, NOW I CAN DDOS FBI SERVERS WITH BILLIONS OF REQUESTS

@eduardocasanova-personal3064 Жыл бұрын

Thanks for the video, makes me appreciate go routines and their simplicity even more :)

@JohnWatsonRooney Жыл бұрын

I couldn’t believe how easy it was when I started learning go

@eduardocasanova-personal3064 Жыл бұрын

@@JohnWatsonRooney preaching to the choir. Even with its rudimentary error handling

@androidmod183 2 жыл бұрын

Nicely done, a Proxy or socks5 "which i believe works with requests" will do the trick bypassing the traffic limit. But how can i implement it in this scenario? Thanks John.

@erenc8377 Жыл бұрын

proxies=proxies :)

@vincentdigiusto9429 2 жыл бұрын

very instructive thank you John, I think that scrapy uses async requests, that's why some scraping jobs can be impressive quick with scrapy

@JohnWatsonRooney 2 жыл бұрын

Yes scrapy uses twisted, which is asynchronous

@BringMe_Back 2 жыл бұрын

Awesome man , I was working with these things Today , I'll try this one too ♥️♥️♥️🙏

@maxjackson6616 2 жыл бұрын

On Yahoo finance, a company's income statement has table rows which need to be clicked on to expand and show the data. Do you know of a way to scrape such rows that doesn't involve using selenium to click on the button to expand the row?

@JohnWatsonRooney 2 жыл бұрын

I haven’t looked for a while but I believe there was a Python package for working with yahoo finiance, yfinance I think? That made it very easy to get all the data

@aabmets 2 жыл бұрын

Thanks, I'm gonna implement this in my "yfrake" package (PyPI).

@Christian-mn8dh 2 жыл бұрын

your videos are as efficient as your code

@mrjt6404 2 жыл бұрын

hey thanks, good info. how to run 'subprocess' Asynchronously, while I dont want to use 'requests.content' to download subprocess.run(["yt-dlp", download_link, "-o", f"{output_dir}/{episode_name}.%(ext)s"], shell=True, stdout=PIPE) Also, What would you prefer Multi-Threading or Async in this case ?

@djangodeveloper07 8 ай бұрын

for me, thread pool executer is best to go. using it from last few years and always gives best results. easy to handle in python standalone scripts or even in python websites with celery.

@acatisfinetoo3018 6 ай бұрын

I was looking for a good explaination of the difference between async and multithreading...so threading is for doing things in parellel and async is for waiting for future tasks to complete but dosn't stall the current program?

@hristijansaveski4231 2 жыл бұрын

Thank you good sir, you are a master at this!! You have helped me land and keep my data scraping job!! Thank you so much, truly an inspiration :))

@JohnWatsonRooney 2 жыл бұрын

Thank you very kind!

@abdullahsiddique7787 Жыл бұрын

In this example when we send 2500 async requests , does each request have the same session or each api call had its own session ?

@dpm_07 6 ай бұрын

no `aiohttp` maintains pool of connection !

@Feel_Sorry 5 ай бұрын

How can we apply same thing in PHP? Currently I am using CURL Multi but it boost up my server utilization. So is there any alternative way to do same in PHP?

@saranpun7192 7 ай бұрын

While using executors, if the capacity of tomcat to process is set to 200 only. What happens after 200 request? Will we have to wait till 200 request are processed by the server? Or, it will pick up some request when call passes from server to db.

@eccentricOrange 2 жыл бұрын

Hey, wanted to point something out: I've often timed my own code too, and I find that print() consistently causes a large delay. So if you're analysing these delays, I would suggest not including a print statement there. Love your content though!! Helped me with a lot of stuff.

@JohnWatsonRooney 2 жыл бұрын

Great tip thank you!

@ericxls93 2 жыл бұрын

Very good vid!! I just finished making use of concurrent futures (based on your previous vid) and it speed up my code considerably! Looks like I have the potential to speed up further 😀. Will making a lot of requests at the same time slow down the source server, thus passing the waiting time to the server?

@JohnWatsonRooney 2 жыл бұрын

It should be fine, obviously you can overload it (ddos attacks work this way) but as we are just trying to maximise our efficiency we can stay within any rate limits or server limitations set. It’s much more useful for getting data from many places at the same time rather than just one

@ahmedelsayed3133 10 ай бұрын

This method puts a lot of pressure on the machine, and to my use case, I want to send more than 170 thousands requests, and when I use ThreadpoolExecutor, I can divide requests to lesser group using max workers' argument. Can that be done using Async & Await?

@JohnWatsonRooney 10 ай бұрын

It sounds like maybe Python isn’t going to be the right language, something like Go or rust might be better for you as they much better built in concurrency models

@ahmedelsayed3133 10 ай бұрын

Which is easier to learn?@@JohnWatsonRooney

@davidl3383 10 ай бұрын

Thank you John

@miguellopez7089 2 жыл бұрын

Awesome vid bro!

@JohnWatsonRooney 2 жыл бұрын

Thanks!

@user-dm9vr6ln3b 5 ай бұрын

can i do the same things with Flask?

@karthikb.s.k.4486 2 жыл бұрын

Thank you for the video. What theme of VS Code are you using please let me know

@JohnWatsonRooney 2 жыл бұрын

Hey thanks, this is PyCharm but the theme is gruvbox which is available for vs code too

@82NeXus 8 ай бұрын

Why does the threaded version take so much longer than the async one, when it should be sending all the requests simultaneously as well? The synchronous one on the other hand waits for each response before it sends the next request. You could have a version that doesn't use any parallel programming facilities in Python but still works in parallel by having the async / parallel stuff handled by a Python package, or handled in an underlying C library. Eg. if you used the (much lower level) socket module and just repeatedly called send (), your requests would all go into the send buffer and the OS would send them while your Python program is waiting or doing something else.

@ADITYAKumar-xi1zt Ай бұрын

Apparently there is something called GIL in python, you might wanna read about that.

@kushagraagrawal7292 Жыл бұрын

Hey I really like your color scheme! Could you please share the theme and color shemes used? Thanks

@JohnWatsonRooney Жыл бұрын

Sure this is gruvbox material and I’m using PyCharm community edition

@terrascape 2 жыл бұрын

hi john, or anyone who knows, I keep getting the error "UnicodeDecodeError: 'utf-8' codec can't decode bytes in position 28855-28856: invalid continuation byte" , do you have a work around for this? the ones that I found on google I can't seem to implement correctly.. much thanks!

@varunvijaywargi5497 8 ай бұрын

Can you please help on how the asyncio would work in AWS Lambda?

@redaoutarid6465 2 жыл бұрын

Thanks for your helpful videos. Please, have you any idea how to avoid dat*ad*ome protection ?

@oparpax 2 жыл бұрын

What if you need to render the content? What would be the best approach in that case?

@JohnWatsonRooney 2 жыл бұрын

Requests-html I believe can do that but I’d have to check

@oparpax 2 жыл бұрын

@@JohnWatsonRooney well that would be a nice topic for a new vid wouldn't i? :D

@danielgarcia1428 2 жыл бұрын

@@oparpax He already did a video on that topic! its called Slow Web Scraper? Try this with ASYNC and Requests-html

@bunnihilator 5 ай бұрын

is the normal request library a blocking library?

@DeepakGupta-qv1yc 10 ай бұрын

Very well explained

@thyagorcarvalho 2 жыл бұрын

That's so great! ow to do this with dynamic payload in a Post request?

@igiveupfine Жыл бұрын

i wish this used a real networked example as i have a real networked API example i'm trying to speed up. i'm already using async and it's literally not any faster. so i don't know if aiohttp will be any better.

@maxjackson6616 2 жыл бұрын

Also, I'm curious whats your day job? Is it related to web scraping?

@JohnWatsonRooney 2 жыл бұрын

It’s not directly no but I use Python everyday for data extraction over our APIs

@techbystorm 2 жыл бұрын

We use asyncio to scrap around 300+ web pages of a site. Because of res = await asyncio.gather(*tasks) the res gets heavier and script stops there. We changed the logic to process 50 pages at a time using asyncio Also, I see you have not used loop = asyncio.get_event_loop(), does this affect the performance?

@user-jj2bx7kt4d Ай бұрын

how to do the same thing using GPU

@jambalaya974 4 ай бұрын

Every devops developers nightmare.

@jamest4027 Жыл бұрын

Hi John, I want to use Asynio and threading in combination. I want to use Asynio for making requests and threading for making calculations. What do you think?

@sebastiangudino9377 Жыл бұрын

You could make a thread that runs a main loop (Inside of which async stuff is happening) while other threads are doing your calculations, so yeah, sure, they can both work But why tho? If you are already making a bunch of threads, why not also use threads for your network request? You are adding complexity to your code base by using both techniques when it seems that your use case would be perfectly fine with just one of them. Making your code slightly easier to maintain So yeah, you can use both of them, but do think about what you are trying to achieve, and if this will actually achieve that

@ZenoModiff 2 жыл бұрын

hello john can you make a video on scrapping world population data website please i tried but failed beacuse the span tag is constandly changing

@andreotako2020 Жыл бұрын

Hello I get this error Event loop is closed

@MichaelSchellerwayne 2 жыл бұрын

Hey John!! Could you make a video about handling cookies with aiohttp? With the ClientSession I can send cookies but they are shared with all instances. I just dont get it how to pass a cookie which is only shared with one website and how to retrieve them. Cookie handling with requests is way much easier, but i really want to use aiohttp because it is SO FAST! :D

@ByteShadow 2 жыл бұрын

What about running async bots that in turn run threads? 🤔

@volin_d 2 жыл бұрын

That's a very neat trick! However is it safe to send so many requests in such a short period of time? I'm learning web scraping and from what I've read your IP can be banned by the website if it detects that all those requests are scripted. That's why in my web scraper I use sleep function to wait from 0 to 2 seconds (randomly) between each request so it resembles more human behavior. But I guess it's not the best solution since it takes about 15 minutes to scrape roughly 1000 webpages.

@JohnWatsonRooney 2 жыл бұрын

Absolutely, if you are working on something that needs to spread out the requests the do so. This is about sending as many requests a possible- if you have 1000 different urls you wanted some data from then having to wait 1 by 1 would be painful, so we can use async to make them quicker. Also learning async technology is definitely worth it as you progress as a developer

@volin_d 2 жыл бұрын

@@JohnWatsonRooney For sure. While I might not use async in this particular case it's good to know about it for the future

@rezanami3925 Жыл бұрын

Dear Ron, I am going to send unlimited fetch requests by javascript code from console to the server. In backend or frontend there is policy for each user to send request with 300 mili second time distance. If I want to send multiple requests in a second, I am blocked. Is there a solution for this issue from your point of view?

@alittleconfused123 Ай бұрын

rotating ips

@mehdinouri5530 2 жыл бұрын

Can you please make a video on how to run a webdriver with headers on google colab? or if you have any tips Thank you I love your vids they really help a lot

@Rbm726 Жыл бұрын

Thanks!

@JohnWatsonRooney Жыл бұрын

Thank you, that’s very kind

@marakeeuh 2 жыл бұрын

Hi Ron! I’m a business engineering student from Belgium currently writing my own thesis. As a part of my research I would need to make my own dataset by scraping Zalando. Would you be open to assisting me with this 1-to-1, ofcourse when publishing my work I would properly reference you and your help. I have tried some things using your very helpful videos, but I’m still mostly very stuck. I look forward to your answer

@renancatan4788 Жыл бұрын

still trying to do? it's possible to scrape this website in a very easy way. I'll be glad to help =]

@kirubababu9255 18 күн бұрын

Anybody summarize here, please

@saurabhjain2437 2 ай бұрын

async for loop would have been more readable…

@sunjayjangam Жыл бұрын

How to send million request?

@tomy7258 2 жыл бұрын

How to login in "jumia"

@oneofthechannelsofalltime Жыл бұрын

More polite title: How to do 2500 handshakes in 2 seconds and disappear. Next video: How to make 2500 people shake hands with each other in 2 seconds, for fun! require 'popcorn' // btw