No video
Web Scraping With Selenium And A Raspberry Pi - All You Need To Know
Рет қаралды 71,220
Күн бұрынThe web and it's websites are complicated. So basic web scraping just won't cut it when it comes to things like logins, forms, and pagination. Well, let's learn how to get what we want using Python, Selenium and a Raspberry Pi.
_____________________________
📲🔗🔗📲 IMPORTANT LINKS 📲🔗🔗📲
_____________________________
• 💻PROJECT PAGE💻 - github.com/gig...
• PREVIOUS VIDEO - Beginners Guide To Web Scraping with Python - All You Need To Know
• quotes.toscrap...
• stackoverflow....
_____________________________
💰💰💰💰 SUPPORT THE SHOW 💰💰💰💰
_____________________________
www.tinkernut....
_____________________________
📢📢📢📢 Follow 📢📢📢📢
____________________________
redd.it/5o3tp8
/ tinkernut_ftw
/ tinkernut
/ tinkernut
@SimSimsTECHcrunch 2 жыл бұрын
The KZfaq legend has returned again!!!!
@UserUnknown07 2 жыл бұрын
Can't imagine the amount of editing this video must have took, woah! Great explanation. Thank you.
@lachlanmoore2345 2 жыл бұрын
Use Explicit Waits when you can instead of the time module, Expected Conditions are great for this.
@timothycain8639 2 жыл бұрын
love this project. you made many aspects of programming with python INFINITELY MORE CLEAR TO ME.
@cyrustakem7993 Жыл бұрын
I miss your videos, i don't know why youtube stopped recommending them, they are highly educative
@johnbushur6080 2 жыл бұрын
Very useful. I came across selenium a while ago but wound up using excel tools instead. I’ll have to give this a try for my next project.
@2mrRB 2 жыл бұрын
Hey John, are you able to use excel tools to scrape websites too? Or do you mean something else? Thanks in advance :)
@johnbushur6080 2 жыл бұрын
@@2mrRB I’ve used Excels web/power query for this in certain cases. Check out Leila Gharani’s channel for some good tutorials. I’ve also written some scripts in VBA to do it as well for specific tasks. That is what I meant by excel tools. Hope that helps.
@jasonbailey9139 2 жыл бұрын
We had a Perl script that we used to scrap data off of a website. They changed the way the login worked and Perl didn't support the new method (OK, it probably does, but I hate working with Perl scripts, so I I didn't bother researching after our consultant said it didn't), so I just made the users start doing the scraping manually. Now I'm tempted to give this a try to start scaping that data again.
@NightRider0101 2 жыл бұрын
Python requests and beautiful soup are the best tools for scraping
@NitishKumarIndia 2 жыл бұрын
This guys belongs to the golden age of KZfaq when the things were simple.
@d-rey1758 Жыл бұрын
where in the video did you mention running this on a raspberry pi?
@twys124 2 жыл бұрын
Great explanation and great video. I just learned about web scraping w BS4 and selenium.
@lukasdegle8313 2 жыл бұрын
Like it a lot! But why don't you use a context handler while writing to files? :)
@domasberulis 2 ай бұрын
what are your rpi specs? Mine 1gb ram RPI 3B takes 3 minutes to launch the browser
@thehoneyseals Жыл бұрын
This made me so happy thank you so much . you have no idea
@JNET_Reloaded 2 ай бұрын
drivers dont work on rpi 5 that well with new supported borowsers so we need real automation without selinium bs any ideas? it should be able to take screenshots and click mouse and use brave browser i got it doing a lot of this stuff but still needs work can you make a video about doing this for rpi 5 using latest brave browser on raspian os debian???
@AS-fj7ox 2 жыл бұрын
Good work dude.. keep it runnin!!
@OnixEdge 11 ай бұрын
@Tinkernut Do you have any tips on how to keep the webdriver updated if you are using the pc and chrome?
@mmuneebahmed 2 жыл бұрын
Awesome, thanks! Will this selenium library also work with any social media websites or do we have to use other libraries in conjunction to selenium?
@mejia414 2 жыл бұрын
Gracias desde Colombia me ayudo mucho tu video
@webslinger2011 2 жыл бұрын
For hiding username and passwords I use config parser to grab from a separate file. What I haven’t figured out is how to use proxies to avoid bot detection. Sorry for the hijack but I need to ask. Anyone with a good tutorial? Thanks!
@NightRider0101 2 жыл бұрын
You can use proxy cycling.
@leader1944 2 жыл бұрын
Proxies would work great to avoid detection if you are sending a large amounts of requests to a site very quickly. However, some sites can detect that you are using an automation software by checking for a string when you send your request with webdriver. This string is $cdc_ and it’s located in the webdriver exe file using a hex editor you can replace $cdc_ with any other string that contains $ at the beginning 3 letters of any kind and then an _ at the end. For example $dog_. Note: Changing $cdc_ only works if you are on chrome otherwise you need to change a different string. Hope this helps :)
@CodingWithBen 2 жыл бұрын
I literally just watched your last video lol. How do I know whether it is allowed to scrape a website or not. Is there an easy way?
@VisesEntei 2 жыл бұрын
Welcome back.
@abrandnewcompany 2 жыл бұрын
Beautiful soup combined with request can do everything what you want, even more than selenium. But I didn't know the NoSuchElementExist Try and catch which is really handy indeed I always use to program it myself a function like that. Thanks!
@paulmagu3054 2 жыл бұрын
Selenium is very useful. Any ideas of running web-scraping on the server side with selenium preferably? (Other libraries in python or Node are welcomed suggestions!) thx.
@dontbelasagna5968 2 жыл бұрын
my csv keep separating the string by characters.. like, the word "the", in csv it is t in one cell, h in the cell next to it, and e in the next one as well..how do i fix this
@AliAli-rj9qb 2 жыл бұрын
if I use bs4 it works fine but with the selenium i get TypeError: zip argument #1 must support iteration. the program is exatly the same as yours so why do i get this error
@spumeeuw430 2 жыл бұрын
I am running into the following issue when trying to install the chromedriver: "E: Unable to locate package chromium-webdriver". Has anybody run into this issue before?
@JasonOBrienThinksHeCan 2 жыл бұрын
Awesome!
@VikashXman 2 жыл бұрын
Thanks man
@papusa9878 2 жыл бұрын
Good video
@AliAli-rj9qb 2 жыл бұрын
sorry i was missing an s in find_elements so now it is working
@Sokar599 2 жыл бұрын
How about puppeteer, isn't that the standard nowadays? Good tutorial als always.
@Tinkernut 2 жыл бұрын
I thought puppeteer was developed for node.js. Is there a python branch too? Selenium is the OG, that's why I went with it.
@Sokar599 2 жыл бұрын
@@Tinkernut Ah yes indeed, I don't often use python I guess. Good to see you're still uploading videos! I used to watch you as a kid all the time. Thanks for educating :)
@100996julen 2 жыл бұрын
I'm planning to do a web Twitter-scrapper program with Python. Which raspberry pi modek is better for it? I want to buy the cheapest that I can. Thanks!
@randomhominid9816 2 жыл бұрын
Why not just use your desktop or laptop computer? A raspberry pi isn't needed but if you want one the rpi 4 with 2GB will probably be enough but maybe get the rpi 4 with 4GB to make sure you have enough memory as browsers tend to use a lot of memory.
@arjix8738 2 жыл бұрын
It's much better to sign up for the twitter API
@jemalguillory 2 жыл бұрын
New drip!
@otmw6726 4 ай бұрын
thanks for not explaining how you found the identifier for the log in button
@gkchimzz28 2 жыл бұрын
nice
@OffGridAussiePrepper 2 жыл бұрын
hahahahaha ur the pun king today :)~
@mefaun 2 жыл бұрын
Yay now I can be Thomas Anderson in the Matrix
@serhiyranush4420 2 жыл бұрын
I am running this script on Windows 7 machine and it works beautifully. However, when running from Thonny, no password prompt appears in the Thonny's console. However, when launching it from the command line window, the password prompt does appear. How can it be fixed for the password prompt to appear in Thonny?
@jyvben1520 2 жыл бұрын
in the console window or did you expect a gui popup window
@serhiyranush4420 2 жыл бұрын
@@jyvben1520 No, I didn't expect a GUI popup window. But I did expect a console prompt, as at 6:42 in this clip.
@Illvidri 2 жыл бұрын
I see the next button and I think "He's just scraping the surface"
@thekevalpanchal 2 жыл бұрын
Hello
@Pod-Z 2 жыл бұрын
Holy shit you listened to my comment
@mfawzi89 2 жыл бұрын
Can I use this code to hack the username and password 😌
@CrjaseMechaEngr 2 жыл бұрын
requests could of done this
@myriadtechrepair1191 2 жыл бұрын
You can scrape my web anytime, pun man.
@4crafters597 2 жыл бұрын
Anyone has a solution to sending the password without including it in code?
@userz111 2 жыл бұрын
Seperated config file Or Use/save-load browser profiles
@woodenbeast9337 2 жыл бұрын
what do you gain by scrapping data? Is this useful?
@yetzt 2 жыл бұрын
data journalist here. yes, scraping is useful if the data you need is not provided any other way. and often times it is not.
@TheOnlyRaichuu 2 жыл бұрын
I'm a freelancer web scraper. There are so many clients. So yes, this is useful. Data is knowledge you can turn into profit. Think about big data companies like Google for example.
@woodenbeast9337 2 жыл бұрын
@@TheOnlyRaichuu It just teaches how to strip our privacy and profit off selling very sensitive data. Running a for profit hack
@TheOnlyRaichuu 2 жыл бұрын
@@woodenbeast9337Why are you asking when you already made up your mind beforehand? What you're saying is absolutely wrong and ridiculous. How does it hurt your privacy when a car dealership wants to get all the data of car listings with their details and price tags to optimize his own pricing? Is anyone affected now in the own privacy? No.
@woodenbeast9337 2 жыл бұрын
@@TheOnlyRaichuu weak comparison
@dudds6699 2 жыл бұрын
Web Scraping with Selenium I know it can be done but its the wrong tool for the wrong job.
@dunste123 2 жыл бұрын
Not enough dad jokes :P
@mohmedbadr1947 2 жыл бұрын
You are late to the party my friend. Most of the website we want to automate or scrap have some antibot
@Tinkernut 2 жыл бұрын
I can see how that may be true for you, but not in general. Most popular websites (twitter, wikipedia, imdb, amazon, youtube, etc) have no such measures. It depends on the website and what they allow. If they have antibot precautions in place, then it's probably not legal to scrape that site anyway. I'm trying to avoid legal issues with this video.
@nibblrrr7124 2 жыл бұрын
@@Tinkernut IANAL, but in the US, *merely violating some corporate website's terms of service is not illegal* _in itself._ See e.g. the EFF's reporting on Oracle v. Rimini 2018 which actually involved scraping. _(Ninth Circuit Doubles Down: Violating a Website’s Terms of Service Is Not a Crime)_ Naturally, I completely understand that you'd want to steer clear of legal issues on your channel, though. (Thanks & keep up the great work, BTW!)
@sarthoknextt5150 2 жыл бұрын
Have you worked as a QA in the past?
@yetzt 2 жыл бұрын
whats up with your sound? it sounds like its out of sync with itself. also i'd recommend going with puppeteer and node if one was more comfortable with js. it just integrates better.
@gmog7857 2 жыл бұрын
Who do you think you are talking to? python experts?
@nibblrrr7124 2 жыл бұрын
Curious people with access to a search engine, motivated to build something they want? :^) If you tell me what it is you'd like to do, what you tried, and where you got stuck or have questions, maybe I can help you or point you in the right direction.
@drewmillett2089 2 ай бұрын
@@nibblrrr7124 Hey I would enjoy some help if you still read these comments. I think I'm getting stuck on pointing Selenium to the correct browser driver path. If I right click on Chrome it shows a path of the executable file but I'm getting webdriver errors when I use this line of code: browser_driver = Service('C:\Program Files (x86)\Google\Chrome\Application\chrome.exe') . I didn't really see how tinkernut came up with the path...
@SeaJay_Oceans 2 жыл бұрын
That is very Edgey comedy...
@astemet 2 жыл бұрын
i got discord bot ready
@Dikkedimi 2 жыл бұрын
dude, your audio is real bad. all over the place.
Family Games Media
Рет қаралды 29 МЛН