Advanced Web Scraping with Puppeteer: Avoid Looking Like a Bot and Pass Authentication!

Рет қаралды 38,557

Жыл бұрын

In this video, we're going to take a look at two puppeteer improvements. First, how can you appear as if you were not a robot? That can be very helpful for avoiding bot protection or captchas. Secondly, how do we get through the authentication of a website? Let's dive right in!
Thanks for watching, I wish you lots of fun implementing these puppeteer tips into your own projects! Remember, some companies do not allow scraping their website, so I advise just scraping your own.. :^)

Пікірлер: 47

@2ru2pacFan Жыл бұрын

Thanks Kevin De Bruyne

@joshtriedcoding Жыл бұрын

Thanks, I like to help a homie out when I’m not scoring goals

@k0iiiii 9 ай бұрын

😂😂😂

@adammaulidikurnia9128 5 ай бұрын

😭😭😭

@egonyuie 3 ай бұрын

😂😂😂😭

@niklasprobst5241 3 ай бұрын

😂

@fitter2boss72 Жыл бұрын

Each video adds something "advanced". Let's continue. Thank you.

@moussaibrahem9 Жыл бұрын

Your videos idea is mind blowing keep going mate

@sebastianruiz8213 6 ай бұрын

Thank you so much! This helped me out on a very important project.

@righttiming 3 ай бұрын

“well… we look like a bot. maybe because we are a bot” 🤣 legend. great video

@samfisher8426 Жыл бұрын

perfect content ,thats what i need to learn ,in case i use it some day in some CTF ;)

@makhmudjonjamoldinov3554 Жыл бұрын

bro, I actually found out that u can set headless to false in the launch options and it works

@codewithguillaume Жыл бұрын

That’s so interesting. I didn’t even know we could have this report as an image. We’ll I think that I’ll spend my weekend working on my bot - however how to host them? Do you have a raspberry pi at home or do you use a regular host online?

@moussaibrahem9 Жыл бұрын

I think because it build on top of nodejs you can host it eny where you want

@ashutoshpatel5030 Жыл бұрын

@@moussaibrahem9 Yeah I too think that you can host that bot just like any other node application we host!!

@dglalperen 7 ай бұрын

First of all nice video ! What can we do about two factor authentication ?

@kingkckc 10 ай бұрын

can you explain how the secret.ts file is structured if we wanted replicate feeding in the login credentials from a different file vs hardcoding?

@bvodola 5 ай бұрын

Got myself unstuck because of this video. Thanks man,

@henriquematias1986 5 ай бұрын

Have you tried doing the same on ebay and try log in? They still detect even if you use stealth!?

@eternl_sunshine22 2 ай бұрын

Hi Josh just wondering how you used cjs modules along with es6 modules, cos i can't seem to make it work

@razshahar7029 Жыл бұрын

thx i search how to fix the err in min 5 ,very helpful

@rodrigodanielss 2 ай бұрын

After 2 or 3 requests amazon fails. Tested the modifying to the plugin and stealth in de video, and still failing the same amount. Gonna have to learn and test with Crawlee.

@GuidoMorabito-ls6xc Ай бұрын

if i use it on linkedin, can it go unnoticed?

@mihaelacostea5783 4 ай бұрын

Would this still work in 2024? Or have big companies came up with the 'defence' already?

@JustinK0 10 ай бұрын

So i guess if the login required to use gmail to login, it wouldn't work because the browser that is opened doesnt seem to allow gmail login api

@Leofmoura87 2 ай бұрын

What's the ultimate solution for resolving captcha?

@fitter2boss72 Жыл бұрын

How to send a form and catch, rename, save a file?

@AbuTaher-vx2oe 11 ай бұрын

It is not working on production server, What can i do?

@thomasdinh2k 4 ай бұрын

I have a question, instead of manual passing authentication, why can't I just login manually and then pass the cookie into the script. Is that harmful or something?