Beautiful Soup + User Friendly + Easy to Learn & Master - Requires Dependencies - Inefficient Scrapy + Efficient + Portability - Not User Friendly Selenium + Versatile + Works well with Javascript - Not Meant to be a Web Scraper - Inefficient

Well done, Sir. You just made it into my "0. Top Resources" Bookmark folder...the competition to get in there is insane and your roommates are very few and far between. It's not what you did in this video per se, it is HOW you did it. Concise, clear, to the point, and not made artificially long to improve your KZfaq revenue. *Make sure* you are advertising to entrepreneurs and I.T. professionals; we have little time (or patience). Thanks again! Well done.

If you liked this video, join the Kite Developer Community on Facebook for access to more resources + support from fellow Python developers. Time to level up! facebook.com/groups/505658083720291

For my use case using Selenium is perfect because I hate dealing with pesky buttons on dynamic webpage.

Let us know what topic we should cover next!

Wow, the quality of the video and the editing is outstanding.

Love this quick video summary. content is perfect and I got exactly what I came for. Some tiny constructive feedback on the delivery would be to speak about them in the same order throughout. 0:39 - Selenium, BS, Scrapy main content - BS, Selenium, Scrapy (best order in my opinion) summary (4:34) - BS, Scrapy, Selenium Keep up the great work!

This dude looks like his birthday is on May 2nd

Mark my word! If you continuously put videos of such great content than you will reach million sub in few years. by the way, I suscribed

nice content

I actually downloaded kite as per your suggestion and am using it now with jupyterlab. It's pretty neat!

Great video! Love how it's concise and to the point. Quick question, can Scrapy be used for scraping sites that use Javascript for dynamic loading too? Or is Selenium the only choice for such a scenario?

Appreciate this video man. Lots of stuff clarified.

Which elements scrapy can fetch??? Like beautiful soup can extract HTML and XML. I have switched to beautiful soup.

so if i want to scrape a large amount of webpages while also activating javascript, i would need to go with Selenium, event though Scrapy would be crawling much faster (out of the box). Would be great to have some kind of tutorial for using Scrapy together with Selenium. I think those too should get along somehow. I guess the only problem is Scrapy is single-threaded and Selenium would Block when its called in this single-threaded environment multiple times, or something like hat.

Great tutorial, thanks for the insight, you saved me a lot of reading ;)

HI, Firstly I want to congratulate you on your new youtube channel and hope that it will be as successful as your product. You are putting in great content, and the dev community really appreciates the hard work. As a future video idea, I would like to suggest "Design Patterns". This would cater to python enthusiasts falling in both ends of the spectrum. Thanks, Cheers!

Hey, did you use Adobe After Effects or some other tool in the starting of video where you were elaborating the agenda of this video?

Thank you for making this video! I'm glad I watched this first before starting to build a scraper.

awesome tutorial, thank you for your video. it is very clear and easy. Also as newby in Shopify eCommerce i am using ""e-scraper"" to scrape shopify stores, all product data from my supplier sites and other sources. It helps me a lot. maybe it helps somebody too. Thank you for your input!!!

Great job on this video. I love how precise it is.

Great content! Thanks! Quick note, maybe center yourself higher in the frame for the camera. Most people have only a little room from the top of the frame to the top of their head. When you center your head in frame as you did, it kinda just makes you look short. Like you could be 7 feet tall but that centering makes you look like a hobbit! Anyway, keep up the great work!

Your video was so easy to understand and help me a lot to know which way to go

Thank you for video! I'm going to check out your web scraping tutorials now!

Summary: Use Scrapy if your data set will be large, else use BeautifulSoup

One could use Helium over Selenium. Helium is built on Selenium but much easier in terms of function calls

This video is so well done!

I hate to be that guy, but Beautiful soup, is not a framework. It's a package/library designed for basic scraping, but its not a framework. In fact, you could, in theory, use BS along with Scrapy as the engine. In comparison, you wouldn't be able to use both Flask and Django together for example, because they are on the same level (Frameworks). Comparing BS to Scrapy, is like comparing Jinja to Django. It doesn't really make sense... even though they both sort of accomplish similar tasks. It kind of feels like you pieced this video together quickly and are giving slightly misleading info.

Amazing Content, New Sub Here!

asyncio & aiohttp + parser of choice 😘

what do you use for your video animations and graphics?

Thank you! That was very helpful!

This is great! Thanks a lot. By the way, could you do a guide on helium? It's a wrapper for selenium but easier to use though I can't get it to work for some reason. Would appreciate a guide video/series.

I came looking for copper and found gold btw great video as always, the edits are really cool~!

Excellent video! Kudos!

I would like a video to explain more about scrapy , because there are few information on internet

Aight bet I’m downloading kite

Very interesting video)) I like it =)

Aside from the learning curve, would Scrapy be the best option?

Hello, your videos very useful

nice content, thnkyou

Is there possible to add some python selenium script backend of our website. If it is not then why and if will then how. I go a website like auto like/auto followers/auto reaction is they use some kind of selenium script

what is better python scrapings or js scraping

Which can be the best for web scraping?

When are you going to post new videos on this playlist? It's been a while 😒

nice video really informative

Great video!!! Can you login to a website like you can with selenium? becasue with selenium you can parse through your userid and password and log into salesforce for eaxample and then scrape accordingly. Do any of the other dependencies or python library provides that feauture? btw this is also the reason i like using selenium.

Nice work 👍

Please tune the video's audio up

Hi! In my application I need to open a web page, fill a form and then click in a button, then get some data that will be loaded in the page. What would you recommend? Thanks in advance

How about supporting of dynamic javascript content? which 1 better?

I want to scrape amazon.com (for monitor arms) and extract prices and shipping weight of each item. Therefore, (it is required to open the page of each item). So what framework you think suits this situation??

If inspect element not allowed for a page,how could we scrsp

Can we use python requests library with scrapy to make post requests to api? I'm done with web scraper but stuck at api

I was extracting text from perticular website it is giving response [500] error ?? anyone knows what it is??

ok, i want to scrape a football teams' forms from a js website, the amount of data is not big. i should go with selenium then? tnx

I would add requests and lxml in adition

@Kite , i want to scrap 1 million records from a website and there are some javascript like some click event require, i know scrapy and selenium, so please tell me which what should i use , scrapy or selenium ?

wanted to install and test kite. i became so sad when found out it needs avx instructions in order to install.

hi, Have you tried Scrapy Splash? for js pages.

nice animations bro

I want to just save image url to hyper link to it

bs4 content returned different than page im viewing for example @t. any ideas

How do we know if the website is using JavaScript or HTML/XML to load contents?

What's not to like about this video?

What is scrappy?

thx

How can I enter a page that has recaptcha2?

Good

Sir what about parseHub ? is parseHub free ?

I hate Beautifulsoup because it never works for me

Do you charge to create a web scrapper?

why dont you use kite

Why do you say bs4 needs dependencies and the others do not? To begin with, scrapy needs twisted, you even mention it yourself!

tldr Simple project? BeautifulSoup Complex? Scrappy Selenium? Better not use it 😁

Dude sounds nervous

I like! Keep it up! Would you like to be KZfaq friends? :)

Good! Keep it up! Would you like to be KZfaq friends? :)

Just use pyautogui lmao