14:45 source venv/bin/activate is for the mac if youre on window ".\venv\Scripts\activate" use this in your terminal

Amazing tutorial, I've only gone through half of it, and I can say it's really easy to follow along and it does work ! Thanks a lot !

Thank you for the time you've put into this tutorial. That being said, you should make clear that the setup is different for windows than Mac. No bin folder for example

this tutorial really needed the code aspect to help make sense of what is going on and fix errors. thanks

The issue we faced in part 6 was that the values added to the attributes of our `BookItem` instance in the `parse_book_page` method were being passed as `tuples` instead of `strings`. Removing commas at the end of the values should resolve this issue. Once we fix this problem, everything should work perfectly without needing to modify the `process_item` method.

I'm starting this course now and very excited! Thanks for the effort of teaching it

Thanks for another great video FreeCodeCamp! This is something I've wanted to spend more time on for a long time with python!!

A wonderful video that we've used as a reference for our recent additions. Your sharing is highly appreciated!

Amazing tutorial, I've really enjoyed watching and it helped me a lot with my project.

I just finished part 7 and want to thanks for the great tutorial!!

at 52:00 you don't need to check for catalogue, you can just follow the url in the tag and it gives me 1000 items

How did u know I needed this course now. 😍😍😍😍 Btw thanks for this free education.

Very clear explanation. Many thanks

Thank you so much for providing this content for free. It's truly incredible that anyone with an internet connection can get free coding education, and its all thanks to people like you!

Thanks Joe Kearney! Nice course of course. You are good teacher, love

Note for Windows users: To activate virtual env, type venv\Scripts\activate

Exactly what I wanted at this moment, Thank you

Thank you very much for this great course. I really learned a lot. ❤❤❤

Thanks alot Freecodecamp for another amazing tutorial ❤️.

Looking forward to this. A mongodb/pymongo section would be nice for data storage though!

Thanks for this crazy course !!!

This is the first coding course I followed up to an end. Nicely taught. Keep it up.

This is so cool! I was able to follow until Part 6 but from Part 7 I couldn't so I will come back in the future after I have basic knowledge of MYSQL and databases. (Note for myself).

This was very helpful, thank you so much for sharing all this knowledge free!

Great tuturial, Thanks a lot!

This is gold for beginners like me. Tks.

thx! very hard to follow, needed a solid knowledge in python

Very practical and helpful video with very detailed explanation!

Thanks for the video. Can Scrapy be used to scrape AJAX responses? Or will Puppeteer / Selenium be more effective?

Finally someone understands what we reaally need and it is published 9 months ago, how did i miss it, i hope this time i will have a Python adventure by not wasting 100's of hours and wasted it for nothing. If it happens again i will curse python and its devolopers. So many changing things, thing can be outdated in seconds and don't know from where to fix it etc. etc. I will hopefully update this adventure.

Hello, thanks for this great introduction to Scrapy However, am wondering in Part 8 for both "User Agents" and "Headers", you set up a boolean var checking middleware_enabled_status 🤔🤔 But, I didn't see it being used anywhere. Personally, I thought it was to be used in the process_request method

I definitely recommend it to everyone 👌👌👌

Thank you, thank you, and once again, thank you!

Overall good video I learn lot of things but I thinks you should discuss briefly about css and xpath selectors. I am facing problem on it

make a course to block the crypto spammers btw thanks for the scrapy course, i was searching for this for a while😃

just in time, thnx tho I didn't knew what i will do for a project i'm working on till i watched the video life saver

Thank you. Great job.

I watched it twice and I think it can be shortened quite a lot and better organized.

oh man , he was just showing me how good his code is !!!!!

How do you guys know I need this for my data analysis project? Fantastic videos, guys! Thank you so much x 5000!

Great content! It helped me a lot to understand some concepts better. 💯

This tutorial is excellent! If you haven't made it already I (and it looks like at least 1 other on this thread) would appreciate a css & xpath tutorial.

For anyone having errors in Part 8 with the fake headers: You need to import this: from scrapy.http import Headers and then in the process_requests function you need to replace this line: request.headers = Headers(random_browser_header)

that's what i need! 👍👍👍

Great tutorial however, I am stuck on part 6 as python is not able to recognize the bookscraper folder as a module and therefore wont let me access the class BookItem to use as a pipeline

Thank you very much for the good work! Really appreciate the tutorial. I need to point out that MySQL I installed with dmg cannot be used with terminal somehow, so I ended up reinstalled MySQL using terminal.

Thank you for these amazing videos, I wonder if anyone can help me, whilst on part 5, for some reason when I exit from the scrapy shell i noticed that I was no longer in Virtual Environment (ie the prefix 'env' was no longer present). When I tried to activate it again using env\Scripts\activate (as Im using windows) it kept giving an error. Has anyone else had this problem? If yes, why does it happen and how can I resolve it?

13:37 creating venv 17:45 create scrapy project 29:31 create spider 33:38 shell

Thanks for such a wonderful web scraping tutorial. Please make a video tutorial on how to download thousands of pdfs from a website and perform pdf scraping with scrapy. In general, please make a tutorial on pdf scraping as well.

Don't know why but Scrapy looks easier to understand for me comparing to BeautifulSoup🤔 Maybe it's just because of teacher or individual mindflow

I had a question. How to split data by table rows for example Name, Price, Description in scrapy

Interesting and useful 😎🤔 video

Great gob thanks !!

Great tutorial!

Great job 🤘

1:24:48 don't forget to remove comas after book_item['url'] = response.url and all others when we add BookItem import. Because i have some values in list instead of string

Great job!

I have mysql installed and can get the version in my cmd, but when I try in VScode, I just get "mysql : The term 'mysql' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again. At line:1 char:1"

Part4 54:07 if you're wondering why the result of 'item_scraped_count' still only 40 probably href is already full url so don't duplicate its domain teach yourself to improvise💪

the course I needed months ago 😭

Excellent video

Amazing video, I learned a lot!!! would you like to do a video how to scrap pages with CAPTCHA? Thank you very mucho for your engagement.

1:34:58 instead of using a lot of if statement use mapping. for example: # saving the rating of the book as integer ratings = {"One": 1, "Two": 2, "Three": 3, "Four": 4, "Five": 5} rating = adapter.get("rating") if rating: adapter["rating"] = ratings[rating] This is not only faster but it also looks clean.

Good day. Thank you for this but i have tried to connect mysql to pycharm on my windows OS to no avail. Any help will be appreciated.

這個音樂真是太美了！喜歡這些東方旋律，讓人心情愉快。 #音樂 #東方音樂 #中國音樂 🎶🌸

Hi, I have a question about venv. Do I need to install a new venv for each part or should I use the venv in part2 all the time?

Golden makadi Your fan 👍🏻

and also make a video on scrapy playwright for JavaScript based sites Thankyou

Nice one!

Thanks!😀

When you save to the database in 2:02:00 ; I had the error because the url was a tuple and 'cannot be converted'. If someone has a similar problem he can just index into the url like this: 'str(item["description"][0])' (instead of the code provided which is this: 'str(item["description"]') in the excute function in the process_item function.

Nice video! Unfurtunelly part 6 has a lot of code without debug, so it's really hard to fix errors. Something is going worng with my code, but i can't identify

Hi, how can i use crapy with pages that are using Ajax? Thanks

They need to make a certification option to whom see all the courses, It'd be so interesting

for windows users: If you get error first type Set-ExecutionPolicy -Scope CurrentUser -ExecutionPolicy Unrestricted -Force and after that type venv\Scripts\activate

Note that I copied the code from the tutorial page for the ScrapeOpsFakeUserAgentMiddleware, and when trying to run it I get the following error: (...) AttributeError: 'dict' object has no attribute 'to_string'. SOLUTION: copy the process_request function exactly as it is in the video, not like in the tutorial page.

Hey guys! Do you have any content on using Puppeteer for JS? :) That would be amazing! Thank you so much for doing what you do

I have a question with part4, in part 4 at first you just scraped one page but later on when we want to have all the next pages and modified it, it still shows me the first page, I'm not sure what is the reason. can you help me with that please? Thank you

I learned the basics of Python and now I want to focus on something to get a job, is web scraping a skill that can get you a job on its own?

GOOD!!!

For PART 8 if anybody is having trouble with the new headers not being printed to the terminal, make sure in your settings.py file that you enable the "ScrapeOpsFakeUserAgentMiddleware" in the DOWNLOADER_MIDDLEWARES and not the SPIDER_MIDDLEWARES.

Thank you for this excellent tutorial! The scraper I had at the end of part 5 somehow only extracted the first book on each page, resulting in only 50 entries instead of 1000. I compared my code to the tutorial code but can't find what I did wrong. I'm wondering if someone had the same issue as me and managed to solve it. Or otherwise, any hints which part of the code might be the cause of this are also very much appreciated. (EDIT: solved the issue - thanks again for making this great tutorial!)

god bless the internet and freecodecamp! thanks !

Great job .. i hope to make a course how to dockrize the scarpy project with postgres DB

I wasted 30 bucks on udemy courses and they are not nearly as good as this tutorial, thanks man

thanks for the tutorial, I have a question, which is the best choice for scraping websites , python or node ?

Thank you for the tutorial, though it is pretty hard to follow for someone - as myself - who's new to Python. Definitely missing some explanation why certain things are done, while you are saying; "then 'just' do this".

Would love some timestamps but thank you for the video!

i want to know how to learn python scrapy

the course is wonderful. It might be more pleasant to listen to if there weren't high sound frequencies in each hissing sound... My sensitive ears caused me to have a headache within a few minutes of the video. Adobe Podcast Enhancer would solve this. It's all very good and could be even better. Thank you for distributing this wonderful course for free

Can someone answer me why many want to use/learn web scraping that need coding while they're many web scraping tools that doesn't need coding to use it? It's because web scraping tools that doesn't need coding usage are limited? Thank you for anyone that answering my question

Please explain in detail how to get the XPaths above. Do they have to be typed manually, or they can be copied from any right-click option??

Just got through 2 out of 5 hours of this web scraping tutorial. Overall, the course is pretty good, but I have to say the teaching quality could be better. The teacher skips over a lot of important subtleties, like Xpaths, which leaves me wanting more depth in the explanations. If you’re new to this, you might want to explore other courses for a more comprehensive learning experience.

🎯 Key Takeaways for quick navigation: 00:00 *Scrapy Beginners Course* 01:51 *Scrapy: Open Source Framework* 03:12 *Scrapy vs. Python Requests* 04:24 *Scrapy Benefits & Features* 05:21 *Course Delivery & Resources* 06:18 *Course Outline Overview* 08:20 *Setting Up Python Environment* 16:38 *Creating Scrapy Project* 20:05 *Overview of Scrapy Files* 26:07 *Understanding Settings & Middleware* 27:13 *Settings and pipelines * 28:22 *Creating Scrapy spider * 30:24 *Understanding basic spider structure * 33:32 *Installing IPython for Scrapy shell * 34:27 *Using Scrapy shell for testing * 36:35 *Extracting data using CSS selectors * 38:23 *Extracting book title * 39:43 *Extracting book price * 40:49 *Extracting book URL * 41:18 *Practice using CSS selectors * 42:02 *Looping through book list * 43:15 *Running Scrapy spider * 47:29 *Handling pagination * 53:52 *Debugging and troubleshooting * 56:12 *Moving to detailed data extraction* Update Next Page Define Callback Function Start Flushing Out Data cleaning process: Remove currency signs, convert prices, format strings, validate data. Standardization of data: Remove encoding, format category names, trim whitespace. Pipeline processing: Strip whitespace, convert uppercase to lowercase, clean price data, handle availability. Converting data types: Convert reviews and star ratings to integers. Importance of data refinement: Iterative process of refining data and pipeline adjustments. Saving data to different formats: CSV, JSON, and database (MySQL). Different methods of saving data: Command line, feed settings, and custom settings. Setting up MySQL database: Installation, creating a database, installing MySQL connector. Setting up pipeline for MySQL: Initialize connection and cursor, create table if not exists. 01:56:31 *Create MySQL table* 02:04:42 *Understand user agents* 02:13:03 *Implement user agents* 02:25:01 *Scrapy API request* 02:26:11 *Fake user agents* 02:27:20 *Middleware setup* 02:33:00 *Robots.txt considerations* 02:40:19 *Proxies introduction* 02:42:34 *Proxy lists overview* 02:52:17 *Proxy ports alternative* 02:52:32 *Proxy provider benefits* 02:53:12 *Smartproxy overview* 02:54:44 *Residential vs. Datacenter proxies* 02:55:27 *Smartproxy signup process* 02:56:19 *Configuring Smartproxy settings* 02:58:07 *Adjusting spider settings* 03:00:23 *Creating a custom middleware* 03:01:21 *Setting up middleware parameters* 03:03:02 *Fixing domain allowance* 03:04:17 *Successful proxy usage confirmation* 03:05:00 *Introduction to proxy API endpoints* 03:06:29 *Obtaining API key for proxy API* 03:07:54 *Implementing proxy API usage* 03:10:36 *Ensuring proper function of proxy middleware* 03:12:10 *Simplifying proxy integration with SDK* 03:13:25 *Configuring SDK settings* 03:14:47 *Testing SDK integration* 03:17:56 *Upcoming sections on deployment and scheduling* 03:21:22 *Scrapy D: Free, configuration required.* 03:21:35 *Scrape Ops: UI interface, monitoring, scheduling.* 03:22:02 *Scrapey Cloud: Paid, easy setup, no server needed.* 03:49:42 *Dashboard configuration guide.* 03:51:21 *Set up ScrapeUp account.* 03:52:48 *Install monitoring extension.* 03:55:24 *Server setup instructions.* 04:00:51 *Job status and stats.* 04:01:47 *Analyzing stats for optimization.* 04:02:42 *Integration with ScrapeUp.* 04:18:05 *Scheduler Tab Options* 04:19:14 *Job Comparisons Dashboard* 04:20:15 *Scrappy Cloud Introduction* 04:21:36 *Scrappy Cloud Features* 04:22:20 *Scrappy Cloud Setup* 04:25:33 *Cloud Job Management* 04:28:57 *Scrappy Cloud Summary* Made with HARPA AI

Sorry, I just want to know how to let scrapeops knows mysql info like (hostname, port, user, password, database) if I don't want to show it in my code directly. I've saved mysql password in .env file at local, and I don't want to push .env file onto github which brings the result that scrapeops can't connect to mysql when schedule run. I also try to add mysql info as key:value arguements when scheduling, but it's not work. Can someone help me to solve it.... :(

is there a second part to this course about scraping the dynamic websites with scrapy?

could you do a video using LLM or LangChain to to webscrape?

"I have a question, does all the change of agents and proxies once we implement them in our code also reflect in the Shell?"

54:36 - 18/05 1:23:16 - 26/05 1:44:19 - 14/06

Hello folks! I'm currently at the part 5 and here is my question. If there is an answer for this in the following parts of this course please let me know. When we export the results into a file, the first book at the file is not "A light in the Attic". Also that is same for me and probably all of us. Is that a problem or what is it? The number of items scraped is 1k, so the scraped item number is matching with the numbers of total books we want to scrape. To be honest I'm wondering how item not coming to output without the order in the site.