Python Requests login and persistent sessions tutorial 🔥: the "Hacker" way

Python Requests login and persistent sessions tutorial 🔥: the "Hacker" way | Python web scraping

Рет қаралды 115,927

4 жыл бұрын

In this Python Requests login to website and persistent sessions tutorial I'll show you how to use the Requests Python library to login to the website, that has a protection and redirects a User after login.
So the main topic of the Python Requests login tutorial is using of Requests Python library to login to the website, using of sessions and cookies.
It's a simple Python Requests Tutorial, that includes the following topics:
- how to login to the website with the Python Requests library only (without Selenium)
- how to bypass the website's protection
- how to scrape data with the BeautifulSoup library
- how to use Python Requests session
I made this tutorial for my subscriber, that asked me to create a Python web scraping tutorial about using the Requests Python library to login to the website, that has a protection from bots.
The Requests is a Python library, that is built upon the Python urllib3 package, and it's a great tool to perform requests to websites.
By default the Python Requests login feature is enabled. We just have to provide login and password data to `auth` parameter. Also Python Requests uses session and cookies.
Also in Python Requests Login tutorial I used the BeautifulSoup library. So at some part this video is a BeautifulSoup tutorial too.
ATTENTION:
in latest Chrome versions, you cant see the form data section, instead you can find the parameters under Payload tab which is next to Headers (many thanks to Nasrullah)
Follow me @:
Telegram: t.me/red_eyed_coder_club
Twitter: / codereyed
Facebook: redeyedcoderclub
✴️✴️✴️ Web Scraping course ✴️✴️✴️
is available via Patreon here:
/ red_eyed_coder_club
or its landing:
red-eyed-coder-club.github.io...
➥ Please ➥ Like ➥ Comment ➥ Subscribe
✴️ More videos about webscraping with Python, Requests, BeautifulSoup ✴️
🔹Python BeautifulSoup tutorial for beginners
• Python BeautifulSoup T...
🔹Scraping dynamic JavaScript/Ajax websites with BeautifulSoup
• Python Web Scraping Tu...
🔹Ebay web scraping with Python and Beautiful Soup: demand research
• Python Ebay Scraping T...
✴️✴️✴️ PLAYLISTS ✴️✴️✴️
🔹Django 3 Tutorial: Blog Engine
• Python Django Tutorial...
🔹Kivy Tutorial: Coppa Project
• Python Kivy tutorial #...
🔹Telegram Bot with Python (CoinMarketCap)
• Python Telegram Bot Tu...
🔹Python Web Scraping
• Python Ebay Scraping T...
➥➥➥ SUBSCRIBE FOR MORE VIDEOS ➥➥➥
Red Eyed Coder Club is the best place to learn Python programming language:
Subscribe ⇢ / @redeyedcoderclub
Python Requests login and persistent sessions tutorial: the "Hacker" way | Python web scraping:
• Python Requests login ...
#python #pythonrequests #pythonrequeststutorial #redeyedcoderclub #requestspython

Пікірлер: 198

@EnglishRain 4 жыл бұрын

This was absolutely mind-blowing. Awesome job, man, SUPERB video. You're a wizard!

@RedEyedCoderClub 4 жыл бұрын

Thank you very much! Glad you liked it.

@abgezocktpfui1134 2 жыл бұрын

PERFECT explained! - especially at the minute 7:55 with the doChallengeResponse part with the mechanism to analyze and reconstruct the md5-coded hash. Good visibility of the font size and the speed of explanation.

@RedEyedCoderClub 2 жыл бұрын

Thank you for you comment, and have a nice day :)

@luccarodrigues781 3 жыл бұрын

Thank you so much for this video! Helped me a lot with my web scraping adventures. Keep up the great work, man!

@RedEyedCoderClub 3 жыл бұрын

Thank you! I glad, that my content was useful for you! Good luck!

@dmitryk1866 4 жыл бұрын

Thanks a lot! Very interesting and useful video. I didn't think that it would be so unusual. Funny conclusion -- "There is no security at the frontend"

@RedEyedCoderClub 4 жыл бұрын

Glad to help, and thanks for you suggestion - very useful!

@gisleberge4363 2 жыл бұрын

Real clever how you solved the "mysterious challenge", great content 🙂

@RedEyedCoderClub 2 жыл бұрын

Thank you very much!

@y.agamer3987 Жыл бұрын

The best video about requests on youtube

@RedEyedCoderClub Жыл бұрын

Thank you very much!

@AK-zh1co 2 жыл бұрын

Excellent video and tutorial. I really like how you walkthrough your troubleshooting and problem solving approach.

@RedEyedCoderClub 2 жыл бұрын

Thank you

@shortcuts9005 2 жыл бұрын

Олег спасибо, хороший материал приятная подача 👍

@RedEyedCoderClub 2 жыл бұрын

Рад, что понравилось

@streamdr1499 4 жыл бұрын

Again, nice, thanks for sharing

@RedEyedCoderClub 4 жыл бұрын

Thanks for watching!

@DeepDiveinUniverse 3 жыл бұрын

The conclusion was good haha

@rationalpi6755 2 жыл бұрын

Great video, thanks!

@RedEyedCoderClub 2 жыл бұрын

Thank you!

@usingvancedplzdontban1128 4 жыл бұрын

Duddddeeee!! You make videos on awesome topics! Thanks A Ton man! Please try and make Docker + Selenium/Firefox/Chrome in the future if you can, thank you!

@RedEyedCoderClub 4 жыл бұрын

Thank you! Ok, I got your suggestion.

@usingvancedplzdontban1128 4 жыл бұрын

@@RedEyedCoderClub thank you dear friend.

@jkf114 3 жыл бұрын

Great video!

@RedEyedCoderClub 3 жыл бұрын

Thank you!

@sven7327 Жыл бұрын

Nice video!

@ozkankurt449 3 жыл бұрын

Red Eyed Coder Club Gerçekten çok dikkatli becerikli bir kanal takipteyim...

@EA-br9ii 2 жыл бұрын

You're amazing. Thank you very much for sharing

@RedEyedCoderClub 2 жыл бұрын

Thank you!

@brunonishimura 2 жыл бұрын

Hi from Brazil!!! great job

@RedEyedCoderClub 2 жыл бұрын

Thank you!

@snopz Жыл бұрын

You helped me a lot thanks ❤️❤️

@RedEyedCoderClub Жыл бұрын

Thank you!

@ArslanFarooqQureshi 2 жыл бұрын

This is great.

@RedEyedCoderClub 2 жыл бұрын

Thank you very much!

@yuniorbarcelochavez6800 3 жыл бұрын

great video! ...

@RedEyedCoderClub 3 жыл бұрын

Thank you very much!

@thegodstudent Жыл бұрын

Thank you bro

@RedEyedCoderClub Жыл бұрын

Thanks for the comment!

@aap204 3 жыл бұрын

Great. I have question about sending a request within a website. For example in this video we want to buy a pair of shoes. When click the buy button we can see sended order in inspect (xhr). Now, i want to learn how to send this order without clicking the buy button in python. Thank you very much

@RedEyedCoderClub 2 жыл бұрын

You need to use Selenium to perform such actions

@response4322 2 жыл бұрын

Can I donate to you??? Because I love this video so much! Wow you helped me solve my issue.

@RedEyedCoderClub 2 жыл бұрын

I'll appreciate for it. My Pay-Pal: molchanov.o.o@gmail.com Or you can make a donation via Patreon

@notAvn 3 жыл бұрын

i love ur english accent

@RedEyedCoderClub 3 жыл бұрын

Thank you!

@shreyaskarsrivastava9386 3 жыл бұрын

Hi, this video very informative! I am scraping a site where there is no visible function which encrypts the password, although under the Network tab, the password value looks to be encrypted. Any suggestions upon how to proceed?

@RedEyedCoderClub 2 жыл бұрын

No, sorry

@Hecticam 2 жыл бұрын

If a website redirects and uses session cookies (for identification) in the main content page, is there any way of using just request library without selenium? I see the cookies already sent in the header of the very first GET request in Network tab, I can't find the way to generate the cookies , please help :(

@RedEyedCoderClub 2 жыл бұрын

In this video I used Requests.Session() class to handle cookies and keep my login stated saved between requests. Whether you need to use just Requests or you need to use Selenium depends on the particular website.

@KRock921 2 жыл бұрын

I liked how you got challenge value, but could you have made your own "challenge" value instead? If the challenge value is set on the front end and posted to the backend and unhashed there, it probably could have been any value.

@RedEyedCoderClub 2 жыл бұрын

No, fits only the challenge value that we get from frontend. I think it uses the same algorithm on backend and frontend.

@Nasrullah0 Жыл бұрын

in lattest chrome versions, you cant see form data section, instead you can find the parameters under payload tab which is next to headers

@RedEyedCoderClub Жыл бұрын

Thank you for the valuable comment! Will add it to the description!

@siddhartha.01 2 жыл бұрын

Thanks a lot man! After 100's of videos I something amazing! Can you scrape Userinfo(like view count followers count etc) of a Quora user without using Selenium or Requests-HTML I'm trying this since past week :(

@RedEyedCoderClub 2 жыл бұрын

Thank you! I'll try it

@siddhartha.01 2 жыл бұрын

@@RedEyedCoderClub Thank you so much, I'm waiting for your try :)

@NicolasMagneTang 2 жыл бұрын

Nice video, One question, for a system of login more complicates like ebay or amazon they have more complicate login protocols with tokems and encripted password. is also possible to get a way to enter with this approach? or it is better to use other method?

@RedEyedCoderClub 2 жыл бұрын

Thank you. I think it's better to use Selenium

@multigladiator384 3 жыл бұрын

Okay, but what can i do if the post request is somehow hidden, when I sniff the traffic with my dev tools and click on the login button, the first request method is actually GET The POST request must happen somewhen before.' The Login Button has an "onLoginClick($event$) in it. I guess I need to find out what happens there to find the POST request

@RedEyedCoderClub 3 жыл бұрын

I didn't get your question. In the video I showed all action I have performed to get all needed hidden fields. For other websites steps can be different. You have to find all hidden fields on the page. Try to disable cache for requests, and check the preserve log option in settings of the Inspector.

@mohammadghorbani2489 3 жыл бұрын

Hi thank you so much for the video. I am wondering if I can login into a broker website and be able to place an order with python? could you please explain briefly how? or if you have already had similar videos just kindly refer me to them. Kind Regards

@albertkunz6730 3 жыл бұрын

Broker pages have higher security measures so it will be more difficult to authenticate. However many broker offer APIs for its users to automate actions. They are often called something like 'trading-api'

@RedEyedCoderClub 2 жыл бұрын

I think it's better to use API or Selenium

@arete_ 2 жыл бұрын

Thanks

@arete_ 2 жыл бұрын

I would love to see more practical examples like this in the future. As you know, almost every web page is different when it comes to logins and protection. These examples are gold for getting practical experience for beginners like me. Love your work. Keep it coming and I shall hopefully tip some more. Happy new year!

@RedEyedCoderClub 2 жыл бұрын

Thank you very much! Happy new year! I'll try to create more videos like this one.

@nygeJIbTB 4 жыл бұрын

Сразу узнал по голосу - Молчанов)

@plxgboi3602 3 жыл бұрын

)

@RedEyedCoderClub 2 жыл бұрын

Yep, it's him

@Romanticoneable 3 жыл бұрын

Thanks for your video man. I subscribed already. I'm trying to do web scraping of Amazon Kindle, but I need to log in first to access the page. I haven't found anything that could help me with this. Could you give me a hand here?

@RedEyedCoderClub 3 жыл бұрын

If nothing helps try Selenium

@gorabramyan 2 жыл бұрын

Hi. Thanks for video. Amazing job. I'm trying to do one with dot freight web site and its so complicated, that they don't have any form in request. Maybe you can give some ideas? Thanks

@RedEyedCoderClub 2 жыл бұрын

Try to check the requests that your browser sends to it. It can help

@Kilogya 3 жыл бұрын

The power of automation got me back into coding. I was a VB.net programmer, but you can guess how boring that got.

@Kilogya 3 жыл бұрын

Thanks for the pin, I've never been pinned so it made my day 🙂

@RedEyedCoderClub 4 жыл бұрын

Let me know what topic I should cover next

@JJBoi8708 4 жыл бұрын

Can you make a video on how to buy products from a Shopify site? That would be so awesome!

@rakeshkumar-jk4lt 4 жыл бұрын

@@JJBoi8708 lol

@TipsyShort 3 жыл бұрын

Hi what if there is no id value

@stealthseeker18 3 жыл бұрын

Can you do web scraping if the website behind cloudflare version 2?

@asifiqbal12141286 3 жыл бұрын

please add the code in description.

@leandrosl86 3 жыл бұрын

interesting, i looking for a solution to made login in a website using request library, however, i would like if know is possible make login session using this method and call Selenium using the same session loged in the website?

@RedEyedCoderClub 3 жыл бұрын

If you have to use Selenium, why don't you want to use it to login?

@leandrosl86 3 жыл бұрын

@@RedEyedCoderClub because in specific situations, I need to press a button to activate a javascript function to load the next page. in some cases, I can get data to navigate over the direct link from the page. I long time ago I searched to resolve this problem over lib request, but I cannot found it any solution for this. Actually, exist any solution to navigate through pages? (about your question) Using Selenium method is more slowly.

@snopz Жыл бұрын

Can you make in future videos how to scrap a website with a recaptcha without solver or selenium for free

@RedEyedCoderClub Жыл бұрын

Thanks for the suggestion, I'll think about it

@snopz Жыл бұрын

@@RedEyedCoderClub ❤️❤️

@dereklee2590 3 жыл бұрын

can u do a video of getting website login details?Lately i been trying to find video how to do that all i know is do something with ip address port

@RedEyedCoderClub 3 жыл бұрын

I didn't get your question. Login details? What did you mean?

@friend1439 3 жыл бұрын

Sir how to contact you? I'm trying to scrape a movies website with your code but it is returning nothing but a pycache folder.

@RedEyedCoderClub 2 жыл бұрын

You did something wrong

@vetrivelr5011 2 жыл бұрын

you simply superb! awesome explanation! Thanks you so much! Can we login if a website require "Soft Token" login? Basically at first time login it asks for "Login", "Password" and it re-direct us to second login page where we need to enter "SOFT TOKEN" (do change every 30 seconds). Once that successfully done it re-direct to Main page/Secure area. Like gmail has 2 way authentication login. Is that possible? Kindly help me! Must have figured this out somehow. Please please help!

@RedEyedCoderClub 2 жыл бұрын

Thank you, Redirects are supported by Requests by default. The main problem here is the getting of tokens. If you will find the valid tokens, then it's probably possible to use Requests for authorization on such websites. It's hard to say something definitely, the task requires a lot of experiments.

@_popbob 3 жыл бұрын

what if a login had a captcha? how do you bypass that?

@RedEyedCoderClub 3 жыл бұрын

You need to avoid getting of captcha. You can solve/bypass simple ones, but not Google's reCaptcha.

@gayashan99 2 жыл бұрын

i found so useful things in your channel. so i subscribed you. i stucked at copying magent link in file cr website. can you tell you me how to do that :|

@RedEyedCoderClub 2 жыл бұрын

I thinks that it can be scraped the same way

@capo8506 2 жыл бұрын

hello, I'm receiving this response trying to login to a website. any help please. "Invalid auth token. Please refresh the page and try again."

@RedEyedCoderClub 2 жыл бұрын

So, you have to provide an auth token. Whether the token is on the page or not - depends on the website you're trying to scrape

@huyhoangnguyen7465 Жыл бұрын

I want to login and dowload file with python ???

@tumtac4544 3 жыл бұрын

how do you web scrape a website that stops your scraping with a cloudfare protection

@RedEyedCoderClub 3 жыл бұрын

What website for example?

@TheSupernatural72 2 жыл бұрын

I tried exact same things and on my website i even have not any challenges, but the only challenge for me is im getting a cloudflare protection "access denied" cause it cant pass captcha from cloudflare. I'm using user agent for that yes. No matter how i tried, couldnt find an answer for that. Could you possibly teach that on one of your videos please? Thanks in advance.

@RedEyedCoderClub 2 жыл бұрын

It depends on the website. Sometimes the method I have shown works, sometime it is not. If the website show you the captcha (Google recaptcha) it's barely possible to solve it. All we can do is to try to avoid of getting captcha. Google's recaptcha is a quite complex thing, that determines whether you're bot or not by analyzing you browsing history and saved cookies. Probably it's better to use Selenium.

@RedEyedCoderClub 2 жыл бұрын

By the way what website you tried?

@moisesmasis1981 3 жыл бұрын

It would be good to know how to make the login in the background and that the system only loads the final page and not that it travels everywhere

@RedEyedCoderClub 3 жыл бұрын

I didn't get your question.

@moisesmasis1981 3 жыл бұрын

@@RedEyedCoderClub It would be interesting if you could make an automatic login that is a background and only show the web after Scrapping and that you execute it in an IIS

@vladislavbadovskyi1666 3 жыл бұрын

Hey, thx for awesome content, can you please make video about ajax buttons like on olx.pl (click the button to view phone)

@RedEyedCoderClub 3 жыл бұрын

Thanks for your suggestion. I'll look at it

@vladislavbadovskyi1666 3 жыл бұрын

@@RedEyedCoderClub I'm already done with it

@RedEyedCoderClub 3 жыл бұрын

Great!

@eminbedir6247 3 жыл бұрын

Hi dude, I also need help. Can you give me a hand?

@jonnikwest6630 3 жыл бұрын

This approach does not seem to work anymore, and I can't figure out why... Has anybody tried it recently or have any ideas?

@jonnikwest6630 3 жыл бұрын

Forget I said anything, I was printing out the wrong variable. Subscribed! :-)

@RedEyedCoderClub 3 жыл бұрын

:-)

@ruiruihuang 3 жыл бұрын

what if there is no login file found in the inspect?

@RedEyedCoderClub 3 жыл бұрын

I don't get your question

@howiewhite4773 3 жыл бұрын

aren't you basically just logging in still but through your code? its not like your hacking in when your still uploading your user and pass in the form data...

@RedEyedCoderClub 3 жыл бұрын

Did you notice these little quotes around the Hacker word? The hacker here means the way of thought to bypass the protection. And nothing more.

@no-user-cw2md9sp9w 4 жыл бұрын

Олег Молчанов

@riam875 3 жыл бұрын

а лол это он?

@no-user-cw2md9sp9w 3 жыл бұрын

@@riam875 да

@RedEyedCoderClub 2 жыл бұрын

Thanks for comment!

@thisismyescape4373 2 жыл бұрын

You are a fucking genius lol😂😂😂

@RedEyedCoderClub 2 жыл бұрын

Thank you :)

@tolulopeayemobola1446 2 жыл бұрын

Hi. Thank you so much for the video. I have a few questions I would like to ask. What email address can I direct my questions to? Looking forward to hearing from you

@RedEyedCoderClub 2 жыл бұрын

If you have a question it's better to ask it here - in comments.

@tolulopeayemobola1446 2 жыл бұрын

@@RedEyedCoderClub Thank you so much for your response. I have a site that I would like to try your techniques on. That's why I wanted to contact you. Anyways, as part of my graduation from a training school, I am working on an aggregator and this would require me pulling data from my own personal banking apps. This is where I was been researching and seeking help. What do you say?

@alexdin1565 2 жыл бұрын

thanks how we can login to gmail using this method

@RedEyedCoderClub 2 жыл бұрын

I'm not sure that is possible to login to gmail using this method. I'm not sure that's possible even with Selenium. Google is too strict. This method is not a silver bullet, but it can work for certain websites.

@abirhasan8315 2 жыл бұрын

pDisk.net Login & File upload tutorial please 👀

@RedEyedCoderClub 2 жыл бұрын

Thanks for you suggestion

@djangopython3051 Жыл бұрын

18:08 Это не weird :) просто надо было использовать `global username` но я думаю ты и сам знаешь)

@RedEyedCoderClub Жыл бұрын

а вот и нет. Откуда там global username?

@djangopython3051 Жыл бұрын

@@RedEyedCoderClub 17:59 на 22 строке ты пытаешься присвоить локальной переменной её же username = username + ':' Python ожидает, что username внутри функции будет локальной переменной. Соответственно, он ищет, где она объявляется в функции. А она не объявлена. Python падает с ошибкой. На что ты и получаешь ошибку UnboundLocalError: local variable 'username' referenced before assignment. Я понимаю что ты хотел изменить переменную username которую импортировал, но пайтон об этом не знает, поэтому ему необходимо указать с помощью global username после 21 строки.

@RedEyedCoderClub Жыл бұрын

@@djangopython3051 Все верно расписал, так и было. Но про global username мне даже в голову не пришло. Ни тогда, ни сейчас.

@iredeyes-4351 4 жыл бұрын

Autobuy script for Realme.com as soon as the stock comes

@RedEyedCoderClub 2 жыл бұрын

Thanks for comment!

@nazarkravchenko989 3 жыл бұрын

Спасибо)

@RedEyedCoderClub 3 жыл бұрын

Thanks for watching!

@abhisheksunda7925 3 жыл бұрын

awesome!! can you make a video on how to scrap " recent submissions" from leetcoode.com?

@RedEyedCoderClub 3 жыл бұрын

Thank you! Thanks for your suggestion, I'll look at it.

@EnglishRain 4 жыл бұрын

At 0:45 you say "without using Selenium webdriver". Any reason for not using Selenium? As per my experience, using a saved Firefox/Chrome profile with Selenium webdriver to gain access is much more simpler & quicker. So why not use it? Is there any advantage of using just requests library?

@libs1213 4 жыл бұрын

There are many advantages, most importantly Requests library is much more lightweight, which means you can run multiple scraping instances without consuming much RAM.

@andkiek 4 жыл бұрын

Selenium it's not quicker, it's so heavy and unpractical in my opinion

@RedEyedCoderClub 4 жыл бұрын

Yep, I prefer Requests wherever it's possible to use, because Selenium is slow... even headless Selenium.

@EnglishRain 4 жыл бұрын

@@RedEyedCoderClub Understood... thanks guys for all the explanations.

@VuNguyen-ni4ex 3 жыл бұрын

How selenium quicker than requests, have u tried both?

@EVILMAJNU 2 жыл бұрын

Sir How To Use Google Recaptcha Same Response Token Again 👍😭

@RedEyedCoderClub 2 жыл бұрын

ReCaptcha is very tough thing. ReCaptcha uses your cookies, your search history and other stuff to determine whether you are a bot or a human. Don't know how to deal with it properly. Obviously the Requests is not the tool you need for it.

@GoodBoyDanny 3 жыл бұрын

Everytime I use selenium or requests on python on this site yeezysupply .com I get banned off the site (can't click any elements) Could you make a video on this?

@amoamo2647 3 жыл бұрын

Not that simple to bot it you will need to inject genned cookies

@mandeelev7221 3 жыл бұрын

Use user agent of your web browser coz some websites detects the bots

@iLLestTv 3 жыл бұрын

Yep. Same shit for me months ago. I was getting closer to the answer, I believe it has something to do with cookies or the user agent. Must manipulate the 2

@amoamo2647 3 жыл бұрын

They have v3 captcha as well so if your not grinning a key for that it won’t go through either

@Christian-mn8dh 3 жыл бұрын

@@iLLestTv i believe it may be you needing to add a user agent within your request. Try this

@vishnudas1791 3 жыл бұрын

Could we inspect these details with any websites

@RedEyedCoderClub 3 жыл бұрын

yep

@vishnudas1791 3 жыл бұрын

@@RedEyedCoderClub which website ??? Am coding in adroid with jupyter)so it's a bit hard could please share where I can inspect these details

@vishnudas1791 3 жыл бұрын

Am asking was there any other option to inspect these details from websites

@vishnudas1791 3 жыл бұрын

I hope you got Bad English😁

@RedEyedCoderClub 3 жыл бұрын

You were asking: "Could we inspect these details with any websites". Yes, you can inspect any website. Consider to buy a cheap or used laptop - it makes you more productive, instead of using Android and Jupiter.

@RedEyedCoderClub 2 жыл бұрын

What video should I make next? Any suggestions? *Write me in comments!* Follow me @: Telegram: t.me/red_eyed_coder_club Twitter: twitter.com/CoderEyed Facebook: fb.me/redeyedcoderclub Help the channel grow! Please Like the video, Comment, SHARE & Subscribe!