Scrapy Basics - How to Get Started with Python's Web Scraping Framework

No video

Scrapy Basics - How to Get Started with Python's Web Scraping Framework

Рет қаралды 33,802

Күн бұрын

Scrapy is a Python framework for web scraping and in this video I will show you the basics of how to start:
* Create a scrapy project
* Use the scrapy shell to find elements
* How css selectors work with scrapy
* Create a simple spider to crawl a website for product information
Code: github.com/jhn...
-------------------------------------
twitter / jhnwr
code editor code.visualstu...
WSL2 (linux on windows) docs.microsoft...
-------------------------------------
Disclaimer: These are affiliate links and as an Amazon Associate I earn from qualifying purchases
mouse amzn.to/2SH1ssK
27" monitor amzn.to/2GAH4r9
24" monitor (vertical) amzn.to/3jIFamt
dual monitor arm amzn.to/3lyFS6s
microphone amzn.to/36TbaAW
mic arm amzn.to/33NJI5v
audio interface amzn.to/2FlnfU0
keyboard amzn.to/2SKrjQA
lights amzn.to/2GN7INg
webcam amzn.to/2SJHopS
camera amzn.to/3iVIJol
gfx card amzn.to/2SKYraW
ssd amzn.to/3lAjMAy

Пікірлер: 86

@pythonantole9892 3 жыл бұрын

Oh my! This channel deserves more subscribers. I scrape a lot of tables in my job but never knew i could use pandas (had never it heard of it) until i saw one of your videos on Pandas. I look forward to more videos on scrapy now that i have the motivation to move away from BS4 and try scrapy.

@JohnWatsonRooney 3 жыл бұрын

Thanks for your kind words I’m glad it’s helped you!

@WildRover1964 Жыл бұрын

a useful start. Followed along and got this working myself (which doesn't often happen when following python tutorials on YT). |Looking forward to finding out now how to get the stuff from page two and then hopefully finding out how to follow links

@nadyamoscow2461 3 жыл бұрын

The best scrapy basic tutorial I`ve seen. Thanks a lot!!

@JohnWatsonRooney 3 жыл бұрын

Glad it was helpful!

@sampatankar1977 3 жыл бұрын

Really lucid, well-judged in terms of content, and excellent videography. Timely too, given what I happen to be doing this week! Thankyou!

@hardwaregenie 2 жыл бұрын

Thanks John for your tutorial. Really liked how easy and approachable you made it.

@celerystalk390 3 жыл бұрын

Great job again John! I've never used Scrapy but now I feel it may be something really useful and powerful. It'd be great if you could do a video comparing the different scraping approaches you've introduced and their scenarios. Thx.

@irfankalam509 3 жыл бұрын

Nice one as always! Hope you would continue this as a series.

@user8ZAKC1X6KC 2 жыл бұрын

Something you note at the 9:23 mark is that you can close the space with a dot (or period). To add a little bit more to that. Regardless of the number of spaces, you only need one period. So close the gap completely and put one dot. I struggled with this for a while, as I had a custom class with 5 spaces (no idea why the coder would do that) in the name and it just never occurred to me that I could just use one dot. None of the documentation in scrapy indicated that. I spent quite a while trying figure that out.

@AmodeusR Жыл бұрын

It's good to learn about css if you're going to use css selectors. The space is closed with a dot because in CSS, when you want to select an element based on a shared class, you write it like "class1.class2". If you were to do "class1 class2" it would mean yout want to select an element that has class2 that is inside of an element that has the class1. To make it clear, we could think of real html elements: "p a" would select any link(a) inside a paragraph(p).

@julz2020 Жыл бұрын

Dude I am loving your videos!! Opening up the wonderful; world of web scraping with these excellent Python tools. Thank you for the content ;]

@JohnMusicbr 3 жыл бұрын

I'm a big fan of your work. Thanks, John.

@JohnWatsonRooney 3 жыл бұрын

Thank you

@sinamobasheri3632 3 жыл бұрын

thanks and nice work John 👌🏻 i was waiting for this in long time 🙏🏻

@sagar318 3 жыл бұрын

Man you're awesome! These videos are so informative and easy to understand, wish you all the success in this world

@JohnWatsonRooney 3 жыл бұрын

Thank you!

@kavehmoradkhani8018 2 жыл бұрын

It tells the educational content very well You're Great. Thanks John!

@theinstigatorr 3 жыл бұрын

Yay! It worked!

@martpagente7587 3 жыл бұрын

Thankyou so much for this John, I hope this will become series.

@JohnWatsonRooney 3 жыл бұрын

Thanks Mart it will

@martpagente7587 3 жыл бұрын

@@JohnWatsonRooney, I hope you can make video also Scrapy-splash approach for scraping dynamic websites by doing some project or sample under this series, thanks!

@edbull4891 2 жыл бұрын

Thank You for this fantastic training. Now I understand where scrapy is all about :)

@engineerbaaniya4846 3 жыл бұрын

Thank John please upload all videos for scrapy

@susannegelarehamiri4497 2 жыл бұрын

Thanks John! Great video.

@mahdi132 Жыл бұрын

Thank you very much your content is awesome

@daniel76900 3 жыл бұрын

as usual...great content...keep on the good work!

@JohnWatsonRooney 3 жыл бұрын

Thanks!

@ALVINMAN452 Жыл бұрын

Thank you, very much.

@litodemesa9699 2 жыл бұрын

You are one of the best!!

@chrissenanayake9891 2 жыл бұрын

Nice presentation!

@stephenwilson0386 2 жыл бұрын

Great intro to Scrapy! Everywhere I've looked people say Scrapy is hard to learn, but frankly this seems more straightforward to me than BS. Maybe that's not the case when things get more complex, but that's just my two cents - maybe you're just better at explaining it? I'm trying to scrape products and prices from Newegg and running into a road bump - I can get the item name and such, but the price is nested in a tag inside a list and finally a div. Any tips on selecting that?

@RenatoEsquarcit 3 жыл бұрын

Appreciated your work!

@d.developer 2 жыл бұрын

yessssss, i'm the 500 liked person!

@JohnWatsonRooney 2 жыл бұрын

Wow thank you!

@kimodataworld5092 2 жыл бұрын

thank you very much wiht your help i did my first web scraping

@JohnWatsonRooney 2 жыл бұрын

That’s great!

@ajayyadav-us8hd 3 жыл бұрын

Hey brother Thanx for the tutorials, can you make a tutorial on other files. eg:- middleware.py , items.py , settings.py And second thing how to use database in scrapy for reading & writing the data.

@JohnWatsonRooney 3 жыл бұрын

Yes will be doing videos on those too

@ajayyadav-us8hd 3 жыл бұрын

@@JohnWatsonRooney Thanks man

@greis790 3 жыл бұрын

@@JohnWatsonRooney An implementation of all the scenarios we use in requests like proxies user agents etc in scrapy framework would be awesome!! Nice tutorial as always!

@NXTTutorials 3 жыл бұрын

Thanks! Very useful!

@mrindia4178 3 жыл бұрын

Thank You!

@mrindia4178 3 жыл бұрын

You are so down to earth, salute to you for providing this type of content for free

@SecurityTalent 2 жыл бұрын

Great

@Chryzsean420 3 жыл бұрын

Just subscribed, Thank you sir .

@JohnWatsonRooney 3 жыл бұрын

Awesome, thank you!

@MohAmuza 2 жыл бұрын

I want to scrapy the product features but it doesn't work properly, I want to get the 4 or 5 features but I get 1 or all features of the page instead, no idea how it's behaving I used this code *response.css("div.f-grid.prod-row ul.f-list.j-list li::text").get()* the code above will print one feature *response.css("div.f-grid.prod-row ul.f-list.j-list li::text").getall()* the code above will print all features of the page while I want to print 4 or 5 depends on the product

@igordc16 2 жыл бұрын

Scrapy seems so intimidating.

@JohnWatsonRooney 2 жыл бұрын

It is when you first look at it, but once you dive in and break it down into parts it will click

@stephennardone5437 3 жыл бұрын

I only recently found your channel but all in all great content! I am however coming across problems with POST requests and selenium is sadly not an option for my project.

@Hugo-pw5ud Жыл бұрын

Thank you!! Almost there but the spider doesnt return the right output. What could be wrong? I do see the 200 scraped items via the shell. Am on Windows.

@JohnWatsonRooney Жыл бұрын

Did you check through the shell response for the items you are after? A 200 can also be something like a captcha page or a blocking page

@victory9654 3 жыл бұрын

Useful video, thanks! You're handsome too..

@JohnWatsonRooney 3 жыл бұрын

Thanks..

@SunDevilThor 3 жыл бұрын

I’m loving these webscraping tutorials. I did get an error though as soon as I tried to use the products variable, such as products.css(‘h3’) I get the error: AttributeError: ‘str’ object has no attribute ‘css’

@Modey3 Жыл бұрын

what is the reason for the venv? are you using a different version of python?

@SabriCanOkyay 3 жыл бұрын

Thanks a lot for the video. I could scrape a website on my first try. I had a problem though. I get this error: raise ExpressionError( cssselect.xpath.ExpressionError: The pseudo-class :text is unknown ... When I changed 'a::text' into 'a::attr(href)' it worked. 'text' was also working in the shell but not in the py file. So, how can I get the texts in the file then?

@prantokhan2303 3 жыл бұрын

scrapy shell 'URL' Doesn't work scrapy shell "URL" Double quote work

@MohAmuza 2 жыл бұрын

I never use quotes

@athenacoding2384 Жыл бұрын

Same for me. Thanks for this comment

@eldarmammadov7872 Жыл бұрын

could you make running scrapy from python script rather from shell

@JohnWatsonRooney Жыл бұрын

Yes you can run scrapy from a script I have a video on it see My channel

@pahehepaa4182 3 жыл бұрын

How do I scrape links from level3 or level4 drop down menus and get output in tree format of all child nodes?

@hardeepbhatti8619 2 жыл бұрын

I really didn't understand the 11:33 part and how you do it btw am new to scrapy . Can you explain it?

@-__--__aaaa 3 жыл бұрын

try with xpath pls

@JohnWatsonRooney 3 жыл бұрын

Sure I’ll use xpath next time

@-__--__aaaa 3 жыл бұрын

@@JohnWatsonRooney thanks ✅👍

@samcamus3000 3 жыл бұрын

Can I use scrapy to scrape JavaScript generated content?

@JohnWatsonRooney 3 жыл бұрын

You can but you need to use the splash extension. I will be covering this soon when I release more scrapy content

@samcamus3000 3 жыл бұрын

@@JohnWatsonRooney 👍👍👍

@artabra1019 3 жыл бұрын

what is difference scrapy on beautifulsoup

@haithemamir223 2 жыл бұрын

But how i can put this data in html

@mohamad5005 2 жыл бұрын

Hi John how can I clear the screen while I am in scrapy shell ? (I use powershell)

@JohnWatsonRooney 2 жыл бұрын

Sure I think typing clear works?

@mohamad5005 2 жыл бұрын

@@JohnWatsonRooney it works before i write the 'scrapy shell order',but after i enter in the response it doesn't work

@muhammadhananasghar3102 3 жыл бұрын

Sir make a video on how to scrape google search results.

@-__--__aaaa 3 жыл бұрын

you should pass useragent in headers

@Don_ron666 2 жыл бұрын

Why does he use a virtual environment?

@nateTheNomad23 Жыл бұрын

Python scraping often involves the use of modules and packages. Once you have multiple python projects, if you don't use a virtual environment, you would have different projects using some of the same packages and modules. If you go to update a package for one project, you would break a different project relying on a previous version of the same package to work properly. A virtual environment isolates packages a modules associated with only one project, so that no matter what other projects use the same packages or modules, they don't interfere with each other. At least that's my understanding.