Hi, my latest course is out now (Professional React & Next.js): bytegrad.com/courses/professional-react-nextjs -- I'm very proud of this course, my best work! I'm also a brand ambassador for Kinde (paid sponsorship). Check out Kinde for authentication and more bit.ly/3QOe1Bh

🎯 Key Takeaways for quick navigation: 00:00 *🌐 Web scraping has been revolutionized by AI, particularly with the latest Vision AI model, making data extraction more efficient.* 01:07 *💻 Manually copying HTML and using Chat GPT for extraction is one method, but OpenAI's API offers programmable solutions for scalability.* 02:16 *🔄 Using Puppeteer with Bright Data's scraping browser helps circumvent website restrictions and rate limiting during scraping.* 05:33 *🖥️ Puppeteer allows for easy scraping of HTML content, but there's a need to manage and clean up the extracted data before analysis.* 08:35 *💡 Extracting only necessary data from HTML can optimize costs when using OpenAI's models for analysis.* 12:17 *💰 Text-based scraping methods can be cost-effective, but they require ongoing maintenance due to HTML structure changes.* 14:49 *📸 Utilizing OpenAI's GPT-4 Vision API enables data extraction from screenshots, potentially offering a more robust solution for complex web scraping tasks.* 17:52 *🖼️ Using base64 encoding allows passing images to models, enhancing data processing capabilities.* 18:49 *💸 Consider cost-effectiveness when choosing between complex HTML-based or text-based approaches for web scraping.* 19:58 *🎚️ Adjusting image resolution can significantly decrease token usage in web scraping, but it may increase the likelihood of errors.* 20:53 *🖼️🔄 Balance image resolution and price when utilizing Vision API for web scraping, as higher resolution images incur higher costs.* 21:19 *🧹 Clean up HTML before web scraping to reduce token usage and ensure accuracy in results.* 22:57 *🤖 Explore advanced features of AI tools, such as identifying clickable elements, to enhance web scraping automation.* Made with HARPA AI

This is such a timely video - i'm doing something similar to resurrect a website from the wayback machine.

Perfect video, thanks

thank you a lot ♥

It's interesting, but what if I want pagination? I will still need to select next button in old way. Is there any other way of doing the pagination?

what an amazing video - like its so niche but so useful

Great video. Some question though. What about hallucinating? How can be sure is not doing it?

Hey man, mind if I ask what programming languages you know other than Javascript/TS ?

Can you create a video how to deploy puppeteer and next js to vercel?

can crewAI do this as well?

amazing

Have you thought about or tried using a local model to scrape, it would save all the costs

how do you handle paginated data?

elegant

Perhaps it will be cheaper on Claude.

How to do this using Braina AI? Braina can run GPT-4 Vision.

This is a great video. But the problem with scraping has hardly ever been parsing the HTML or maintaining the parsers. The biggest problem is efficiently accessing websites that actively try to block you by gating their content being a login or captchas. Then comes IP blocking (or worse data obfuscation) if you Scrape their website in a large volume.

How a full stack dev work with AI?

I am interested in creating a price comparison website featuring approximately 10-20 shops, each offering around 10,000 similar products. Unfortunately, these shops do not provide APIs for direct access to their data. What would be the most efficient approach to setting up such a website while keeping maintenance costs reasonable?

I am scrapping (dropping html) with python code with selenium (aprrox 60,000 articles) and later creating vector embeddings for Llama 3 and asking it to write article for me.

First