Web Scraping with GPT-4 Vision AI + Puppeteer is Mind-Blowingly EASY!

  Рет қаралды 20,947

ByteGrad

ByteGrad

Күн бұрын

👉 Thanks to video sponsor Bright Data! Get $10 free credit: brdta.com/bytegrad1
Hi, I'm Wesley. I'm a brand ambassador for Kinde (paid sponsorship): bit.ly/3QOe1Bh
👉 NEW React & Next.js Course: bytegrad.com/courses/professi...
👉 Professional JavaScript Course: bytegrad.com/courses/professi...
👉 Professional CSS Course: bytegrad.com/courses/professi...
👉 Web development roadmap 2024 & 2025: email.bytegrad.com
👉 Email newsletter (BIG update soon): email.bytegrad.com
👉 Discord: all my courses have a private Discord where I actively participate
⏱️ Timestamps:
00:00 Intro
00:48 ChatGPT for HTML
02:01 OpenAI API
04:08 Puppeteer
04:28 Avoid restrictions (Bright Data)
05:11 Get HTML with Puppeteer + Proxy
08:35 Processing HTML
10:00 Give HTML to OpenAI
12:19 ChatGPT for scraper code
14:41 Vision API
19:40 Vision API pricing
21:12 Vision vs Text pricing
22:52 Future of web scraping (AI + Bright Data)
#webdevelopment #programming #coding #reactjs #nextjs

Пікірлер: 28
@sidouglas
@sidouglas Ай бұрын
This is such a timely video - i'm doing something similar to resurrect a website from the wayback machine.
@nomicgaming5730
@nomicgaming5730 Ай бұрын
thank you a lot ♥
@reidevanson181
@reidevanson181 Ай бұрын
what an amazing video - like its so niche but so useful
@ByteGrad
@ByteGrad Ай бұрын
Glad you liked it
@hxxzxtf
@hxxzxtf Ай бұрын
🎯 Key Takeaways for quick navigation: 00:00 *🌐 Web scraping has been revolutionized by AI, particularly with the latest Vision AI model, making data extraction more efficient.* 01:07 *💻 Manually copying HTML and using Chat GPT for extraction is one method, but OpenAI's API offers programmable solutions for scalability.* 02:16 *🔄 Using Puppeteer with Bright Data's scraping browser helps circumvent website restrictions and rate limiting during scraping.* 05:33 *🖥️ Puppeteer allows for easy scraping of HTML content, but there's a need to manage and clean up the extracted data before analysis.* 08:35 *💡 Extracting only necessary data from HTML can optimize costs when using OpenAI's models for analysis.* 12:17 *💰 Text-based scraping methods can be cost-effective, but they require ongoing maintenance due to HTML structure changes.* 14:49 *📸 Utilizing OpenAI's GPT-4 Vision API enables data extraction from screenshots, potentially offering a more robust solution for complex web scraping tasks.* 17:52 *🖼️ Using base64 encoding allows passing images to models, enhancing data processing capabilities.* 18:49 *💸 Consider cost-effectiveness when choosing between complex HTML-based or text-based approaches for web scraping.* 19:58 *🎚️ Adjusting image resolution can significantly decrease token usage in web scraping, but it may increase the likelihood of errors.* 20:53 *🖼️🔄 Balance image resolution and price when utilizing Vision API for web scraping, as higher resolution images incur higher costs.* 21:19 *🧹 Clean up HTML before web scraping to reduce token usage and ensure accuracy in results.* 22:57 *🤖 Explore advanced features of AI tools, such as identifying clickable elements, to enhance web scraping automation.* Made with HARPA AI
@imranhrafi
@imranhrafi Ай бұрын
It's interesting, but what if I want pagination? I will still need to select next button in old way. Is there any other way of doing the pagination?
@hishamazmy8189
@hishamazmy8189 Ай бұрын
amazing
@hellokevin_133
@hellokevin_133 Ай бұрын
Hey man, mind if I ask what programming languages you know other than Javascript/TS ?
@juliushernandez9855
@juliushernandez9855 Ай бұрын
Can you create a video how to deploy puppeteer and next js to vercel?
@RobShocks
@RobShocks Ай бұрын
Have you thought about or tried using a local model to scrape, it would save all the costs
@user-vo8kk8dv8l
@user-vo8kk8dv8l Ай бұрын
elegant
@Garejoor
@Garejoor Ай бұрын
can crewAI do this as well?
@Lars16
@Lars16 Ай бұрын
This is a great video. But the problem with scraping has hardly ever been parsing the HTML or maintaining the parsers. The biggest problem is efficiently accessing websites that actively try to block you by gating their content being a login or captchas. Then comes IP blocking (or worse data obfuscation) if you Scrape their website in a large volume.
@binhtruongdac2861
@binhtruongdac2861 Ай бұрын
That’s why you need smth like Bright Data, yes, it’s not free unfortunately
@subhranshudas8862
@subhranshudas8862 Ай бұрын
how do you handle paginated data?
@binhtruongdac2861
@binhtruongdac2861 Ай бұрын
You just need to use the URL with page number in query params then run for loop to request multiple html page
@LifeTrekchannel
@LifeTrekchannel Ай бұрын
How to do this using Braina AI? Braina can run GPT-4 Vision.
@dmytroocheretianyi7577
@dmytroocheretianyi7577 24 күн бұрын
Perhaps it will be cheaper on Claude.
@amadeuszg1491
@amadeuszg1491 Ай бұрын
I am interested in creating a price comparison website featuring approximately 10-20 shops, each offering around 10,000 similar products. Unfortunately, these shops do not provide APIs for direct access to their data. What would be the most efficient approach to setting up such a website while keeping maintenance costs reasonable?
@Braincompiler
@Braincompiler Ай бұрын
Make it like the other comparison sites and provide an upload for CSV, XML and so on or YOU provide the API for them so their shop systems can push the data ;) Crawling by yourself is the last option and could be made with XPath and stuff.
@amadeuszg1491
@amadeuszg1491 Ай бұрын
@@Braincompiler Yes, but in this case store needs to send me the csv, xml file with their products. What if they dont?
@Braincompiler
@Braincompiler Ай бұрын
@@amadeuszg1491 Yes of course. If your comparison site has a benefit for them be sure they will.
@abhisycvirat
@abhisycvirat Ай бұрын
I did this 6 years ago, scraped each website and compared the price using SKU
@amitjangra6454
@amitjangra6454 18 күн бұрын
I am scrapping (dropping html) with python code with selenium (aprrox 60,000 articles) and later creating vector embeddings for Llama 3 and asking it to write article for me.
@richerite
@richerite 4 күн бұрын
Do you have a GitHub link? What did you mean write article
@UserAliyev
@UserAliyev Ай бұрын
First
@semyaza555
@semyaza555 Ай бұрын
2nd
The magical amulet of the cross! #clown #小丑 #shorts
00:54
好人小丑
Рет қаралды 25 МЛН
Let's all try it too‼︎#magic#tenge
00:26
Nonomen ノノメン
Рет қаралды 54 МЛН
didn't want to let me in #tiktok
00:20
Анастасия Тарасова
Рет қаралды 12 МЛН
Bro be careful where you drop the ball  #learnfromkhaby  #comedy
00:19
Khaby. Lame
Рет қаралды 17 МЛН
The Biggest Issues I've Faced Web Scraping (and how to fix them)
15:03
GPT-4 Vision API + Puppeteer = Easy Web Scraping
56:25
Unconventional Coding
Рет қаралды 164 М.
GPT-4o is WAY More Powerful than Open AI is Telling us...
28:18
MattVidPro AI
Рет қаралды 191 М.
Web Scraping with ChatGPT Mentions is Mind Blowing!
8:42
The PyCoach
Рет қаралды 24 М.
How to use AI for Web Scraping with Excel and Google Sheets
5:19
Peter Mangan
Рет қаралды 10 М.
GPT4V + Puppeteer = AI agent browse web like human? 🤖
24:48
AGI: solved already?
22:11
John Koetsier
Рет қаралды 14 М.
The magical amulet of the cross! #clown #小丑 #shorts
00:54
好人小丑
Рет қаралды 25 МЛН