A Guide to Web Scraping with Node.js

Рет қаралды 370,191

4 жыл бұрын

Learn how to build a web scraper ⛏️ with NodeJS using two distinct strategies, including (1) a metatag link preview generator and (2) a fully-interactive bot for Instagram. fireship.io/lessons/web-scrap...
1. Build a simple link preview scraper with Cheerio.
github.com/cheeriojs/cheerio
2. Build an interactive Instagram scraper with puppeteer.
github.com/GoogleChrome/puppe...
#javascript #nodejs #webdev
Take the quizzes 🤓
iOS itunes.apple.com/us/app/fires...
Android play.google.com/store/apps/de...
Upgrade to Fireship PRO at fireship.io/pro
Use code lORhwXd2 for 25% off your first payment.

Пікірлер: 416

@oleksandrisaryk4728 4 жыл бұрын

Me yesterday: Hmm...there is no API available, well, need to learn how to scrap with node. Fireship today: A Guide to Web Scraping with NodeJS Well done.

@chicken_teriyaki 4 жыл бұрын

No one like this comment, it has achieved greatness

@BearologyLabs 3 жыл бұрын

@Killian Malik Tried plenty of methods, this one seems to be working!

@lizzagarcia6405 2 жыл бұрын

@PaleSaturn 4 жыл бұрын

This is one of the best programming channels on KZfaq. Love the production on your videos!

@PawelPolskiPL 4 жыл бұрын

I love it! It’s the most useful video on KZfaq for me, ever! Today all day I was making app using puppeteer, I didn’t know it’s so easy, thank you as always for great video. Now I’ll be saving over 40 min every day at work 😂👍

@MrDots99 4 жыл бұрын

I've been doing this for that past 2 days at work and you upload a video today haha , fortunately I got it working but its always nice to see you upload !

@actualmortgage7122 3 жыл бұрын

Thanks for this upload. I got into a little bit of web scraping a couple years ago but ran into a lot more complexity than I was expecting for a tiny little side project. Now that I know a lot more, and JS, hoping to add this to the toolbelt.

@theblackharted 4 жыл бұрын

Your content is top notch and so well done!! Easily my favorite programming channel

@arisweedler4703 4 жыл бұрын

I love it! :D such well-made videos. I appreciate how you explain the problems that packages solve instead of just trying to peddling hype

@remoteworkboard 4 жыл бұрын

You must be a mind reader. I was struggling with my link preview generator all week! Thank you soooo much.

@alexanderf7008 6 ай бұрын

I only start understanding things better after watching your videos. It's such a great explanation and never boring.

@armaandhanji2112 4 жыл бұрын

HUGE fan of your Node videos. Thank you so much! Best channel on KZfaq.

@ninanordbo 2 жыл бұрын

Love your content! Well done, my favorite youtube resource. Love that you get straight to the point.

@boriskrstic 4 жыл бұрын

This video is literally like you listened to me today what I was wondering about. Thanks for the puppeteer part! :)

@Fireship 4 жыл бұрын

Awesome! The topic came from a suggestion in the slack channel last week :)

@firaskudsy 4 жыл бұрын

Thanks for those amazing videos... u r covering the full full stack 👍

@steaklover575 4 жыл бұрын

Been here for a long time, your content just gets better and better!

@curiousnrd 4 жыл бұрын

Fantastic video! I’m excited to test this out. Thank you for sharing this information.🙏🏾

@jimmymac601 4 жыл бұрын

As a long time subscriber, you need to make tutorials every day. Fantastic work.

@Fireship 4 жыл бұрын

I wish I could, my idea list is massive.

@TheGejr 4 жыл бұрын

Just as I'm building a news-scraper you make this amazing video! You're using the exact same packages I am :^) Great video as always!

@calebkopp4174 4 жыл бұрын

Very insightful. Appreciate all the work that's put into the videos

@ambarmutha8504 4 жыл бұрын

🔥ship is AWESOME! Every time I have a project on my mind I find something useful to steal from this channel.

@justafreak15able 4 жыл бұрын

In an open source world there is no such thing as stealing.

@bartub5369 4 жыл бұрын

Great content! I have previously made a scraper for a discord bot with cheerio before, but never really fiddled around with puppeteer or anything similar to it earlier. Thanks for the ideas :)

@SmartWizzard 4 жыл бұрын

Man you are awesome, all your videos are latest, useful and very informative. I very much happy that I have subscribed to your channel.

@diabolo1 4 жыл бұрын

OMG I’ve been looking for all the web for resources on WebScraping and suddenly your video appears! Well done!

@lyricalstudio7845 4 жыл бұрын

All your videos are amazing!! Keep up the good work and we will do our best to support your channel. Cheers!! :-)

@albertchung7641 3 жыл бұрын

This channel's videos are all so cool. You rock!

@tanujvyas6124 4 жыл бұрын

Thank you for this video .I was looking for guidance for web scraping .This video prooved to be a great source.

@f3lixadam 4 жыл бұрын

Hi Jeff, thank you so much for your videos! We are building a startup and without your tutorials and explanations, I don't know if we could have ever gotten such a great insight how the mechanims work, especially for Ionic/Firebase. Much Love from Germany

@Fireship 4 жыл бұрын

Wow, that is awesome! Keep me posted on your progress :)

@pepperpeterpiperpickled9805 Жыл бұрын

@@Fireship If I didnt think you were an AI, I'd want to buy you a beer

@iykazorji8171 4 жыл бұрын

This is some quality content yo! Thanks for this!

@greatsuccess4734 4 жыл бұрын

I have been waiting for this video for so long thank you so much

@Oxcorp 4 жыл бұрын

Love your videos man, best quality!

@Autoscraping 5 ай бұрын

An outstanding video that has been a valuable reference for our newcomers. We sincerely thank you for sharing!

@sebastiancuk7004 4 жыл бұрын

Im using puppeteer in my work to make short videos testing the app automatically (feels awesome). Also really cool video keep the good work !

@raph6709 Жыл бұрын

Dude...this is exactly what I needed... why are you so helpful

@camerenisonfire 4 жыл бұрын

Always such interesting and well produce videos. Thanks, Fireship.

@AaronJack 4 жыл бұрын

So many things could be built around puppeteer, and the await syntax makes it so easy to reason about. Also seems like this is a perfect use-case for cloud functions vs having a dedicated API server. Great video.

@igotapochahontas 4 жыл бұрын

This is a really good video. Very clear explanation and simple

@stevewitman 4 жыл бұрын

Interesting topic and it was also helpful seeing a firebase cloud function being set up.

@Tizmo-tt9ry 4 жыл бұрын

when you say as hackers i feel so cool

@Fireship 4 жыл бұрын

Haha, web scraping always feels very hacky.

@anonwithamnesia 4 жыл бұрын

@@Fireship Which font do you use?

@Fireship 4 жыл бұрын

@@anonwithamnesia Fira code

@StingSting844 4 жыл бұрын

Haha yeah its hacky and cool. Once you start to make money off scrapped pages you will sued though

@vaishnav4750 4 жыл бұрын

😊yes

@ShivamGupta-nz7dy 4 жыл бұрын

When it comes to Web Scraping, most will talk about Python. Felt very happy knowing Web Scraping can also be done easily using JavaScript. Thanks a lot.

@softwarelivre2389 4 жыл бұрын

Javascript is the native environment for web scraping. Not only it is asynchronous, but also Javascript was created to be used on the web. To see people using interpreted, slower python for that is just sad. Not that there is something wrong with python, but it has other purposes. JS is much better for scraping pages generated with React, for example. On python, you can, at best, use a bot to simulate a user control. With JS, you can evaluate the DOM and then retrieve your data without the need of emulating a real user, and that is very powerful.

@alokrawatt 4 жыл бұрын

Thanks for your video, I have all of your videos. Loving it.

@almostanengineer 4 жыл бұрын

I love the fact your using Insomnia for your rest requests, when I started working with API, I was told to try Postman, and found it overly complicated everything, then I found Insomnia, so much cleaner and easier to use.

@anuraghazra4772 4 жыл бұрын

I agree

@jeandrepentz5011 4 жыл бұрын

That puppeteer library looks neat, never knew about it

@yejielwahnich6136 4 жыл бұрын

Link preview I wonder if I will ever need this. Said jokingly knowing they talked about it just 1 week ago. Awesome content !!

@Fireship 4 жыл бұрын

I get my best ideas from Slack :)

@yejielwahnich6136 4 жыл бұрын

@@Fireship not for nothing we love/hang around slack ;)

@adante407 4 жыл бұрын

This video is great! Just subscribed. Fantastic channel, amazing content. 👍

@drowmik 4 жыл бұрын

That a cool detailed video in a very short time about scrapping...

@alinandrei9006 4 жыл бұрын

Thanks, understood what I came for from the first two minutes.

@bharathravi7820 4 жыл бұрын

Was looking forward to this, thanks a ton!

@randith87 4 жыл бұрын

Just getting into Node, and Json. Having a lot of fun! Definitely a challenge! 😎💻

@G3Number 4 жыл бұрын

Your tutorials are top quality !!

@OniasdaRocha 4 жыл бұрын

you just got yourself a subscriber, great content my man

@gavinmurphytheperson5446 4 жыл бұрын

Thats pretty cool. Don't have a use case yet to use this yet but I can imagine how useful it could be. Great video as usual!

@tworizki 4 жыл бұрын

It is. Imagine you could scrape the entire net and let people browse your scraped pages using keywords. Oh wait...

@T3P 4 жыл бұрын

I really love watching your videos ❤️

@zdravko6t5 4 жыл бұрын

Very interesting video. Learned so much from this video. Didn't know you can do such idea using Instagram.

@iova666 4 жыл бұрын

wish i had this vid a few years ago. you just gained a new sub :)

@iddrissraaj6737 4 жыл бұрын

Great Video, informative as always

@anonwithamnesia 4 жыл бұрын

Great Video! I love your videos. Keep it up bro!!

@coolaydalena 4 жыл бұрын

Ooohhhhwwww.. More Ideas about cloud functions please... 🔥❤️

@williamragstad 4 жыл бұрын

Amazing video! Love your content!

@mateiadriel353 4 жыл бұрын

the puppeteer support for firefox is awesome!

@trusterzero6399 4 жыл бұрын

As a programmer there is a lot of stuff that I kinda want to look in to but I never really get the chance to. These videos are helping me hard with that

@viveksoundrapandi 4 жыл бұрын

Never new about puppeteer. Excellent one

@juancamiloq1 4 жыл бұрын

Awesome content bro. Congrats!

@jlai383 4 жыл бұрын

I like this kinda demo so much!

@SouthPeter98 4 жыл бұрын

Great video, as always!

@htky 4 жыл бұрын

Perfect timing, I just started a project that will involve scraping 👌🏻

@akarshbarar1492 4 жыл бұрын

Hey one of the thing I have been searching from last 4 months..

@zarefgamz2515 4 жыл бұрын

You saved me a lot of time to find the name of a tool It has been more than 3 days searching for that tool

@DisasterSPA 4 жыл бұрын

Amazing video as usual!

@firstprincipleslearning 4 жыл бұрын

Great videos as always !

@bjerz23 4 жыл бұрын

Your videos are just so good

@yallayeho1238 4 жыл бұрын

Very interesting topic! Thank you!

@alexgogan1617 4 жыл бұрын

I'll be having such a scrape time with this!

@xit 4 жыл бұрын

That Tshirt Looks SICK!

@AngrejKumar 2 жыл бұрын

this is awesome man. Thank a lot!

@RoshanKumar-yr1jf 4 жыл бұрын

Thanks for the great video... I always wanted to do something in this domain...I was just shifting to python scrapy tool... Thanks again for awesome video..😃😃

@DarkNenyk 4 жыл бұрын

Thanks to you I got into Angular

@firestoreprojects6202 4 жыл бұрын

Amazing as always.

@gauravdasgupta3663 4 жыл бұрын

Everytime I share some url in my LinkedIn posts..it loads the preview so beautifully...and I was always thinking how do they do that... U made it so easy.. now I can do it too... Thanks a ton

@troykoss 4 жыл бұрын

Great stuff, Jeff!

@MagnusVestergaard 4 жыл бұрын

Web scraping is a great thing to know, thanks!

@sachinprabhuk6241 4 жыл бұрын

Awesome videos. Keep up the awesome work

@tekforge Жыл бұрын

Thanks for the insights!

@ryansamarakoon8268 4 жыл бұрын

This is great! I want to try make like an app that uses face recognition to try find pictures of you where you were tagged or not, but I was too late to start using the API. This now gives me hope that it's possible

@will_abule 4 жыл бұрын

Thanks Jeff this can be extended for ssr and seo 😁

@scottmccartney1518 4 жыл бұрын

Awesome vid, really appreciate it!

@milesmeow 4 жыл бұрын

A great video for adding a tool to our web Swiss Army knife.

@vinukurian389 4 жыл бұрын

Exactly what i was looking for 👨‍💻

@Dumbastic 4 жыл бұрын

Your videos are awesome!

@mksoftwaresolutions9303 4 жыл бұрын

Another great video. Thank you.

@rockwoodfrank1517 4 жыл бұрын

Very useful. Thank you!

@aravind.a 4 жыл бұрын

Good explanation with example, Jeff.

@fakesingh3235 4 жыл бұрын

I was looking for this only!!

@GavinLon 4 жыл бұрын

Very interesting video and very well presented. Thank you.

@adrienlucbert2769 4 жыл бұрын

Clean and interesting video, as usual ;)

@alexcubed4270 4 жыл бұрын

Great video yet again

@lifasibiya4810 4 жыл бұрын

Thanks again for the video Jeff. Last year, I was using a library called HtmlAgilityPack to achieve some of this with C#. You've made it quite simpler. YOU DESERVE THE FIRST FLYING CAR MAN👍🏾