🛑 How to Scrape UNSCRAPABLE data! (super simple!) Node.js + API

Рет қаралды 53,442

Жыл бұрын

Click here to follow along: brightdata.grsm.io/webscraperIDE
__
⭐ Sign up for my Full Stack Developer Course: www.codewithania.com
⭐ Check out my IDE here and get 1 month free: jb.gg/get_webstorm
⭐ New to code and none of this is making sense? Watch my '12hr+ KZfaq Coding Bootcamp' in which you will learn HTML, CSS and JavaScript Fundamentals completely from scratch. It's on my channel and its 100% free.
⭐ In most videos I use Tabnine as my A.I autocompletion tool. You can download it for free here (I get no commission from this link, but am in a partnership): bit.ly/tabnine-top-tool
⭐ You can get a blockchain domain with my affiliate link here: bit.ly/get-a-crypto-domain
⭐ If you would like to buy me a coffee, well thank you very much that is mega kind! : www.buymeacoffee.com/aniakubow
⭐ Sign up for weekly coding tips from my newsletter partnership: bit.ly/JS-tips
You can also find me on:
Twitter: / ania_kubow
Instagram: / aniakubow

Пікірлер: 106

@Hobbitstomper Жыл бұрын

The best solution imo is NodeJS + Puppeteer + puppeteer-extra-plugin-stealth plugin. It's free, doesn't rely on any 3rd party APIs and works 100% to avoid cloudflare blocking and other captchas. You can even log into any website, even if it uses OAuth for Google, Facebook, Amazon, Microsoft, Twitter, Apple etc.

@hazemelbatawy1242 Жыл бұрын

great job I followed your steps and really it was fantastic, I am a data scientist and you impressed me. God bless you and if you need anything like Machine learning I am working on algorithms.

@PySnek Жыл бұрын

You're the best Ania! Thank you so much!

@SamLinnett Жыл бұрын

Nice one Ania - this is really great

@shrikantjha5630 Жыл бұрын

Yes, I have also used cheeriojs with react native as an experiment and it worked well.

@marekr.9339 Жыл бұрын

Pozdrawiam z Polski i życzę dalszych sukcesów w rozwoju kanału!

@aniakubow Жыл бұрын

Dziekuje 😍😍

@paulthomas1052 Жыл бұрын

Thanks - very useful as usual :)

@NAHChannel Жыл бұрын

Great video! I've had some better experience scraping using xpaths instead of classnames in sites which dynamically generate the classnames. But it seems to go down to the content being scraped. Scraping using CSS selectors seems to be faster also.

@richardmasters2045 Жыл бұрын

Thank you so much Ania 🥰

@Crakkovia Жыл бұрын

Good Work Annia!

@iankila5618 Жыл бұрын

Ty for these tutorials!

@Erwin_t Жыл бұрын

WOW amazing tutorial! I love your style and your approach. I am starting web development. I want to learn Vanilla JS your way. What is the best practice to learn and retain the methodology of JS? Please help :)

@srishimalah9561 Жыл бұрын

@Code with Ania Kubów,Hi, your video of the battleship is unavailable.Can you please look into it ? Because your video is the part one and the part 2 & 3 three is working. I am trying to study the game logic and it will be very helpful if you can re-upload your video.Thank you.

@dystopian_1 Жыл бұрын

See you in 3 days, Mother of Dragons.

@aniakubow Жыл бұрын

See you there 🐉👑

@coderizer Жыл бұрын

great video, keep up the good work

@DevMadeEasy Жыл бұрын

Great Content, as usual, thank you so much for sharing it with us, I know how hard is to build a project then edit it, post... Thanks🙏

@irobot8297 Жыл бұрын

would love to see your computer setup, your desk, keyboard chair etc :)

@joelayoub2774 Жыл бұрын

Looking forward to this.

@2ru2pacFan Жыл бұрын

Hey Ania, do you know how to scrape websites blocked by Cloud Flare? X

@thefeelingofunfair4052 Жыл бұрын

Do you have your series 7 ?

@ROVAKAN Жыл бұрын

At the last video there was axios + express module , but i tried it on react result was CORS errors. Maybe this video is going to tell about that kind of errors and maybe about proxy set ups.

@aniakubow Жыл бұрын

I hope it solves your issues too :)

@hassaneoutouaya Жыл бұрын

Thank you so much !

@miguelbcn Жыл бұрын

Im trying to do the same with twitter to get the tweets from any user, and it seems imposible. Could you help me?

@gaia2933 11 ай бұрын

Although there are many software solutions for automating and extracting data from a website, using NodeJS and its library ecosystem remains the most flexible option offering endless possibilities.

@silversolver7809 Жыл бұрын

Ania, fyi in case it might affect the algorithm: "unscrappable" should only have one 'P'-ie "unscrapable" :) Also, not sure it's a common word.

@aniakubow Жыл бұрын

You make a very good point! Thanks for having my back 🙌🙌🙌. What is a better title do you think?

@silversolver7809 Жыл бұрын

@@aniakubow You're welcome :) Ideally put the most relevant words first, as it's likely algos will regard those as more important than later words. So instead of say "Make your videos better on KZfaq", you should have "KZfaq videos-improve yours". Also, it's usually better to use positive words rather than negative-eg "You will win" > "You can't lose". So: Scrape ALL Data Scrape EVERYthing Scrape and Catch ALL Data Emphasize ALL and EVERY, because that's the unique point of this video-if everything is in CAPS, then nothing is emphasized.

@halowarstier3147 Жыл бұрын

I am stuck on npm init, not sure how to follow instructions. Please help

@thunde7226 Жыл бұрын

That was great Ania.....................take care ........................:) bye

@AMoktar Жыл бұрын

Amazing you are ❤

@philipbengtsson2186 Жыл бұрын

After viewing this video it would be interesting to see what we can do to prevent others from scraping our own website projects. 😅

@aleksandrkobelev8868 Жыл бұрын

I love you so much! You are the best!)

@aniakubow Жыл бұрын

you are !

@godswillhycinth9809 8 ай бұрын

Thanks for the video, can this also scrape out Instagram HTML content?

@ClimbHighWithAI 4 ай бұрын

I need to do it with more than 5000+ products and also need description and price and etc how can I do it

@joseluisperez5137 Жыл бұрын

A little notecheck at 7:08, use the -D flag when installing nodemon, nodemon is just for development on this example

@joseluisperez5137 Жыл бұрын

Another sticknote, like the documentation says, is not a web browser instance, it just takes the html to interpretate and do the job, so, if we do this stuff on websites that don't do server side rendering at all, will be missing some information since maybe it's loaded by external sources, like multiple scripts, external call apis, etc.

@producdevity Жыл бұрын

This sticknote is more than just a note. It’s the difference between pulling your hair out and understanding right away why some values are populated and some are not.

@drucifer6 Жыл бұрын

Amazing content! I'd be curious how to scrape/store data in a database and use that for my own frontend.

@Bot-kl1gs Жыл бұрын

Hey ania can you also include the part where you can store the fetched data in a database(Like mongodb) and then show the user. it would be a great help OwO OwO

@avenazpk Жыл бұрын

Supabase is best choice

@hussainnajaf332 Жыл бұрын

The question of which language do you use and why do you use it ؟

@golgappayadav1864 5 ай бұрын

Can u make myntra scrapper video

@mgusa9372 Жыл бұрын

Killer look, light pink that's definitely you. Scrap 'UNSCRAPPABLE' data yeah I'm in, I'll be back, spoken in an Arnold hillbilly German accent. Love your stuff GO Ania.

@kacperkepinski4990 Жыл бұрын

How did you get that accent?

@BreakfastCupNoodles 9 ай бұрын

The problem with managed one is the cost. For custom one, you can pay for as low as $19/month for 100,000 pages. It's also not hard to scale.

@techwithulises Жыл бұрын

No lo quiero, lo necesito

@screendice4107 Жыл бұрын

Thanks mam

@js_models Жыл бұрын

I wish there was an npm Ania command, because she is the total package. 😉

@1A_B_C1 Жыл бұрын

Queen 👸

@lowlufi Жыл бұрын

Lovely

@albaraasaad4498 Жыл бұрын

Thank you for the great content. I have a request because I've been searching all over to find a good explanation on how to scrape pages that have a load more button - NOT DIFFERENT PAGES - using Cheerio and Puppeteer. I can scrape a page when it's auto-loading when scrolling down but still couldn't make it by clicking the load more button😭. Thank you.

@qualitytransportation Жыл бұрын

Just click it with puppeteer then load with cheerio

@albaraasaad4498 Жыл бұрын

@@qualitytransportation I know that it should click, but whenever I try it's not working. I mean the puppeteer will not click the load more. I did navigate the click button and but I don't know why it's not working.

@startupshorts Жыл бұрын

Hi Can u please explain how to scrape email from LinkedIn

@aniakubow Жыл бұрын

I think the video should help with that :)

@christiandanielmoralesagui4659 Жыл бұрын

See you soon Teacher

@urosjovicic3988 Жыл бұрын

I came here to learn, instead i fell in love :D

@jesusmoran1356 10 ай бұрын

didn´t work to me

@screendice4107 Жыл бұрын

Mam I am waiting.Why you did not list this video on top?

@aniakubow Жыл бұрын

Oh I am not sure! Weird 👀

@ashwinr8317 Жыл бұрын

hi ania

@aniakubow Жыл бұрын

hiya!

@VesuviusAntaria Жыл бұрын

Hi Ania! 🙂🌸🏵🌹🌺🌼🌻🌷

@VesuviusAntaria Жыл бұрын

Thank you for your kiss! You have made my day! 🙂🌺

@---fq2kd Жыл бұрын

ты супер !

@luissosa7685 Жыл бұрын

Hola 👋

@gileneusz Жыл бұрын

ale się produkujesz ;) scrapowanie to ciężka sprawa....... sam ostatnio bawię się w diffbot'a

@pastuh Жыл бұрын

At such times I would say... AI must understand what to scrape.

@balakumar.n4891 Жыл бұрын

How to scrape Formula 1 data ?

@aniakubow Жыл бұрын

This video should help I think :)

@rachest Жыл бұрын

I need you as my technical partner

@briandsouza7854 Жыл бұрын

I think putting a premiere 24 hours would be better. This long wait feels annoying!

@StephenChapman Жыл бұрын

It doesn't have to feel annoying. Just tap/click the notify button, then put it out of your mind and move on to thinking about literally anything else in the world.

@christian-schubert Жыл бұрын

I'll second that. I'd even go as far as calling all those announcements years in advance spam, literally made me unsubscribe from this channel. Now, that's not to say that the content itself isn't of high quality. Ania is a real gem - I keep checking back occasionally. 👍

@desi_vlogs005 Жыл бұрын

👍

@__--JY-Moe--__ Жыл бұрын

🥳

@abhishekkaith1686 Жыл бұрын

I personally use jsdom don't know why lol

@michaelallen1154 Жыл бұрын

I'm here to learn. 🙄

@jeanmi8184 Жыл бұрын

how to scrape your ❤

@joelapablaza7722 Жыл бұрын

So.. if u want to scrap a dynamic web just go the sponsor of this video.... really?

@atlantic_love 9 ай бұрын

Yeap, she's just getting subscribers off her looks, and using these stupid sponsors as her "content". I disliked this video, and another one. In watching the previous one I couldn't figure out whether she just can't type or she doesn't really know what the heck she's talking about.

@roostermarques3583 Жыл бұрын

I think someone is trolling off your comments.

@socar-pl Жыл бұрын

>How to scrape data >Use paid service that sponsr this video ayyyyyyyyyy lmao

@aniakubow Жыл бұрын

I show two ways to do it so you can choose :)

@i-am-your-conscience Жыл бұрын

I am almost embarrassed to admit on how much easier it is to learn such stuff when your teacher is just smokin' hot :D besides being an amazing teacher already, dont get me wrong :)

@dystopian_1 Жыл бұрын

Titanic was lost in your bright eyes.... lovely, lovely you...

@Qasim6 Жыл бұрын

😜

@trolley2327 Жыл бұрын

I think there is a whole generation of programmers in love with her :))

@AADJgroup Жыл бұрын

😱😇

@wgalloPT Жыл бұрын

I wish so much I had a girlfriend just like you...smart, beautiful and a coder!!

@illegalsmirf Жыл бұрын

Using PHP or Perl?

@yobi3d Жыл бұрын

Как обычно, все "очень просто"! ) Как её смотреть то? Стояк мешает )

@atlantic_love 9 ай бұрын

Nothing but a sponsor video.

@richardmasters2045 Жыл бұрын

My cyber girlfriend the smartest woman I know. You have my undying love, respect and devotion 🥰 I can't wait seriously on the edge of my seat 🤓

@user-mh1km3mv7z 8 ай бұрын

um, update the old video so that it actually works then do this christ id like to do your projects but id ont know this node.js technology for new versions!

@aniakubow 7 ай бұрын

You can change the version of node.js to the one I am using in the video. Just check the package.json for the version :)