Want Faster HTTP Requests? Use A Session with Python!

Рет қаралды 47,405

3 жыл бұрын

This is why I think you should use a http session when web scraping with python. It comes with many benefits and lets us access more features within the requests library, the most common and used Python library for http requests. In this video we look at the connection pooling that is allows us to access to speed up our code when sending requests to the same server. This is perfect for scraping data or accessing APIs.

Пікірлер: 94

@celerystalk390 3 жыл бұрын

Another great video covering an important topic for efficient web scraping. Thank you John!

@pr0skis 3 жыл бұрын

Another fantastic vid! Short, succinct and very useful! I'll be applying this sessions thing to some of my code very shortly! As always, thanks for the great content.

@JohnWatsonRooney 3 жыл бұрын

Thank you!

@cetilly 3 жыл бұрын

Oh Jeesh. First day using the Session and it made a HUGE difference in performance (14 seconds for what used to take 2 minutes!). Thanks for covering this topic so well.

@JohnWatsonRooney 3 жыл бұрын

Great to hear!

@cetilly 3 жыл бұрын

Best vid on Requests.Session I've seen. Practical and very helpful

@GordonShamway1984 3 жыл бұрын

As always excellent work! Thank you so much to share your insights

@nikhil182 Жыл бұрын

Thank you so much! I was looking for quick tutorial on keep-alive type of connections using requests library and this video solves my problem.

@ugurdev 3 жыл бұрын

The wonders of internet in Australia, I am getting 46 seconds without and 38 with. haha. Good stuff, thanks John.

@tubelessHuma 3 жыл бұрын

Nice short video to elaborate the efficiency of session object. 💖

@multigladiator384 2 жыл бұрын

Thanks again, John! Workwise I am "implementing" two different APIs'. They have rate limits and there is there case where I need to slice the input args and send multiple requests. This is how I will do it.

@nurshah816 2 жыл бұрын

I was pretty amazed with Session performance, thanks a lot.

@katherinebaker3220 2 жыл бұрын

John, you are really cool. Your videos help me a lot!

@0xfsec 3 жыл бұрын

Thank you so much for this Awesome tips John!

@avkngeeks 2 жыл бұрын

i really appreciate your lessons, the best on youtube

@abhijeetbonde8635 3 жыл бұрын

i was reading about sessions for 2 days and still couldn't understand the concept and just 7:15 minutes and i got a whole new level of knowledge... Thanks for the video buddy... really helped me...

@JohnWatsonRooney 3 жыл бұрын

No worries glad to help!

@easyshopping7264 3 жыл бұрын

Thanks for your great effort, you are the teacher, with my regards , waleed

@KhalilYasser 3 жыл бұрын

An awesome trick that will make difference certainly.

@darrenhuang2600 3 жыл бұрын

amazing video, helped out a lot

@i701Dev 3 жыл бұрын

Great video. Thanks for this.

@potatopc Ай бұрын

this is life saving!!!

@thatguy6664 3 жыл бұрын

Another good one...cheers!

@JohnWatsonRooney 3 жыл бұрын

Thank you! Cheers!

@SkySesshomaru 2 жыл бұрын

Wow. Thank you bro!

@Klausi-uq4xq 3 жыл бұрын

Nice to know..such a difference..

@mehdismaeili3743 Жыл бұрын

Excellent.

@JesusTorres-bt2eb 3 жыл бұрын

Amazing, you should do a video for scraping on a cloud flare website, but thank you

@rakshithrajesh3938 2 жыл бұрын

The Best Web Scraping Channel on KZfaq. He Teaches Each Topic so Clear and is very easy to Follow. Highly Recommend this Channel to anyone who wants to Learn Web Scraping the RIght Way. Keep it up John

@JohnWatsonRooney 2 жыл бұрын

Thank you very kind!

@alien9940 Жыл бұрын

Super helpful tip

@JohnWatsonRooney Жыл бұрын

Thanks! Check out my latest video if you liked this one much more on using the session

@Niki14741 2 жыл бұрын

thats very helpful, thanks

@flimsy1417 2 жыл бұрын

great video :)

@Achiesamablog 3 жыл бұрын

Maybe it is stupid question, but please bear with it as I am quite new to this. Will requests.Session() keep the session alive if I provide cookies and user agent (headers) to my requests, for example s.get(url, headers=headers, cookies=cookie) ? Will it still speed up my process? thank you for kind souls who will answer this question in advance. And thank you for the video too!

@primestarchannel7221 2 жыл бұрын

Great video, i tried it worked like a charm. Thank you. But when I do session.close and retry it says connection reset by peer. How to completely close the session?

@quangjoseph8287 2 жыл бұрын

Hi, i could get crsftoken from the cookies with Session for logging in Memrise website in 2021. But I don’t know why it doesn’t work the curent days. Could you please give me any advise?

@tanzimat2039 2 жыл бұрын

Hey John,, thanks for the beautiful contents firstly

@JohnWatsonRooney 2 жыл бұрын

Hey! Generally no, you should always use a session, it won’t speed it up significantly enough to cause blocking

@Ervilhav10 2 жыл бұрын

Nice video dude! Now my default way is session.get() instead of requests.get() hahaha

@HuskyTales2023 3 жыл бұрын

Hi can requests do sign-ups? I know it can do logins but what if we can send the sign up data thru post params.. besides the recaptcha token us a problem. Can you do a video to pass that kind of payload with recaptcha token

@lakshminarasimhanadimoorth3945 Жыл бұрын

Can we apply this to optionchain of Indian Stock Exchange(NSE) so that I can transfer the data to excel.

@campbat5712 Жыл бұрын

Thanks, I went from 500 requests every 87 secconds to 500 requests every 33 secconds which is like the maxinum amount I can do considering the ratelimit

@erdemgunal13 3 жыл бұрын

is there a way to get data from website has cloudflare ddos protection

@welcometout Жыл бұрын

I want to exit or close my session after my code complete its work. How to close the session at end of the code execution

@Automatic-show Жыл бұрын

John, in your opinion, which method works faster, session or concurrent.futures or asyncio ???? speed is one of the most important factors in this work.🤔🤔🤔🤔🤔🤔🤔

@rtxmax8223 2 жыл бұрын

the requests library is working slow in my laptop (80 secs), when i run the same code it works fast in macbook (0.5 secs). What could be the issue ?

@DM-py7pj 3 жыл бұрын

If you use gettitle_session(s , x) isn't s at this point a session object? So is there a reason for not passing it as an argument to the function v referencing globally?

@JohnWatsonRooney 3 жыл бұрын

Yes it is, we give the function out session object to use - you need to reference it outside the function though so every request uses the same session to make it cleaner and a bit faster

@aogunnaike 3 жыл бұрын

Very interesting

@maciekpaciarski9343 3 жыл бұрын

Hi .when can one expect new live web scraping with python ?

@novianindy887 Жыл бұрын

Does the session literally means creating application session in server then append session-cookies in every of its get request? or it means TCP session via Keep-Alive connection?

@novianindy887 Жыл бұрын

I mean the requests.Session()

@axelamoe Жыл бұрын

Can you show us how to handle required cookies by a site?

@dalefixter 2 жыл бұрын

when using pytest, can the session be persisted in the test, when the session had been created in a conftest fixture?

@JohnWatsonRooney 2 жыл бұрын

My initial thoughts are yes, I can see that you could do so within the same session but it’s not something I’ve done before, my experience with pytest is limited

@glenn8781 2 жыл бұрын

I love how you just slipped in 'pip install requests' as a text overlay in the beginning of the video 😂

@JohnWatsonRooney 2 жыл бұрын

😅 thanks

@indhumathi7391 2 жыл бұрын

Can session used for 504 Gateway timeout error in Python API?

@JohnWatsonRooney 2 жыл бұрын

No a 504 is a server error not a client one so it won’t solve that issue

@pahadivillager5354 3 жыл бұрын

Nice video! How do we close this session without using the "with" keyword?

@JohnWatsonRooney 3 жыл бұрын

Session.close() I believe!

@islamibrahim4735 3 жыл бұрын

Very nice, can you make video on how to scrap LinkedIn Jobs based on profession and salary using scrapy

@tushargupta5890 9 ай бұрын

My code improved by 75 %

@hlrn4141 6 ай бұрын

Would you still recommend session over httpx?

@JohnWatsonRooney 6 ай бұрын

Httpx has its own session object too

@manikandanmanickam9433 3 жыл бұрын

Pls do scraping to for the Delta airlines website

@PolSenserrich Жыл бұрын

Don't you have to close the session every time??

@gouemoregis195 3 жыл бұрын

Here we go

@mr.strange7002 2 жыл бұрын

HELP !! I am using pythons request lib and I want to get data from an API .. so I write down a program and run it on my local computer it just run perfectly .. but when I try to like host the code in web like heroku or in replit it responds with status code 200 but not providing data inside the api... I tried to change the user agent as a browser agent but it doesn't worked.. can anyone help me out..

@JohnWatsonRooney 2 жыл бұрын

A 200 response could still be a captcha page or similar - try checking to what information is coming back, however it’s likely being blocked as your now on an IP shared with lots and lots of other people

@mr.strange7002 2 жыл бұрын

@@JohnWatsonRooney It's return a json string with two keys like message : ok & Type : ok in webhost but when I try the exactly same code with same bearer token and headers on local computer it provides the entire json file with all the data ... How can I fix this ...

@JohnWatsonRooney 2 жыл бұрын

Is it an api with an actual key or from web scraping?

@mr.strange7002 2 жыл бұрын

@@JohnWatsonRooney It's an API with authentication token and headers

@JohnWatsonRooney 2 жыл бұрын

That’s odd. Im not sure, sorry!

@nooral-firdaus7144 2 жыл бұрын

Unfortunately, it did not work for me. With or without session, the running time is the same... :-(

@JohnWatsonRooney 2 жыл бұрын

Really? Ok fair enough. To go faster you’ll need to learn about async and await

@deepak2950 3 жыл бұрын

Why my session is taking more time than normal

@aiisnice1453 3 жыл бұрын

Or you can do that: from concurrent.futures import ThreadPoolExecutor def FetchAllListing(page): print(page) r = requests.get(f"rei.com/c/backpacks?json=true&page={ page }") with ThreadPoolExecutor(max_workers=30) as e:#Threads e.map(FetchAllListing, range(50) )#Amount of pages to scrap

@kenkelvin4023 3 жыл бұрын

Nice

@bestpoppers7369 3 жыл бұрын

Good idea

@Christian-mn8dh Жыл бұрын

ly bro

@kenkelvin4023 3 жыл бұрын

Use pypy3, it’s what I use along with golang

@bunyaminsahiner9060 3 жыл бұрын

hello. thank you so much for this wonderful tip. I would like to use this session. My .py similar to your 'Web Scraping with Python: Ecommerce Product Pages. In Depth including troubleshooting' video .py How I can add in this codes? there are 2 requests. first for find links, second for products data.