Web Scraping 101: A Million Dollar Project Idea
Рет қаралды 532,172
Күн бұрынWeb Scraping is one of the best projects you can work on with real, legitimate potential to make you a TON of MONEY ($$$)! It's the ability to collect real-time data about travel, eating, commerce, healthcare, real estate, you name it. Learn about this multi-billion dollar industry and how to tap into it!
Sign up for @BrightData here and get $15 in FREE credit! brdta.com/techwithtim.
BrightData Scraping Browser: brightdata.com/products/scrap...
Project Code: github.com/techwithtim/Price-...
Data Collection Project Ideas: • Data Collection Projec...
🎬 Timestamps ⏱️
00:00 | Web Scraping
00:19 | The Potential
00:56 | The Idea
01:38 | The Process
02:12 | The Problem
02:47 | The Solution
03:36 | Million Dollar Project Demo
06:47 | Code Walkthrough
11:16 | BrightData Setup
12:14 | Next Steps & Ideas
💻 Master Blockchain and Web 3.0 development today by using BlockchainExpert: 🔗 algoexpert.io/blockchain (Use code "tim" for a discount!)
💻 Accelerate your software engineering career with ProgrammingExpert: 🔗 programmingexpert.io/tim (Use code "tim" for a discount!)
◼️◼️◼️◼️◼️◼️◼️◼️◼️◼️◼️
👕 Merchandise: 🔗 teespring.com/stores/tech-wit...
📸 Instagram: 🔗 / tech_with_tim
📱 Twitter: 🔗 / techwithtimm
🔊 Discord: 🔗 / discord
📝 LinkedIn: 🔗 / tim-ruscica-82631b179
🌎 Website: 🔗 techwithtim.net
📂 GitHub: 🔗 github.com/techwithtim
One-Time Donations: 💲 www.paypal.com/donate?hosted_...
Patreon: 💲 / techwithtim
◼️◼️◼️◼️◼️◼️◼️◼️◼️◼️◼️
⭐️ Tags ⭐️
- Tech with Tim
- Programming
- Web Scraping
⭐️ Hashtags ⭐️
#techwithtim #programming #coding
@TechWithTim 11 ай бұрын
Let me know any web scraping project ideas you have! Sign up for @BrightData here and get $15 in FREE credit! brdta.com/techwithtim.
@AR-rg2en 11 ай бұрын
Thanks for all the videos man, I appreciate your work.
@jgalvan09 11 ай бұрын
you know this is exactly why Twitter limit the amount of content people can look at lol use AI bot to do web scraping for you lol we're going to abuse social media now to the point every platform will have to set limits lmao
@nevilleachi6888 11 ай бұрын
i am new to web scraping but i have been researching on how to make an ai webscraper that can automatically open the ispect page of a website and extract the necessary elements given to it by a user. the user only needs to specify what he wants from a website and the scraper does all the work. it can be particularly helpful for newbies
@AR-rg2en 11 ай бұрын
@@nevilleachi6888 If you could make such a thing I'll use it 😂
@Pabbzzo 10 ай бұрын
@@AR-rg2en🎉😊😢
@flin4557 11 ай бұрын
So this is basically an elaborate commercial
@twentytwentyeight 10 ай бұрын
Just an anecdote, but when I first got into a data role I built a competitor price web scraper for my company. At the time it was supposed to be a show and tell for my boss in case python work came up in the future. It worked well enough for the business to utilize for beating the comp to vendors on price. We could keep tabs on them at a quick enough pace to adjust quotes and win. Didn't always work out but being able to say a python project netted three extra contracts is really cool
@moeyali123 9 ай бұрын
Nice, I’m in a data role
@gustavonovakoski4867 2 ай бұрын
I am currently developing this exact project with Power BI as dataviz tool. Currently have nearly 60 websites scraped and nearly 3300 urls daily updating. Kinda sad because everything I do is local and dont have the infrastructure to put it in the cloud or run from VMs (also I dont really know how to get more deep than this lol) The company is a multinacional and they DO NOT have market intelligence, it is bizarre
@voitkampff 10 ай бұрын
Great video! Thank you for talking about web scraping as an IT subset. What most people don't know is that it's extremely hard to build your career around it. I have been running my own web scraping software house for several years now and getting clients is more difficult than if you build apps, create websites, or do virtually anything else. People and companies simply don't have enough awareness what access to real time data can do for their businesses. Creating this awareness is actually the biggest part of my job. Writing code is just a formality afterwards.
@samtx 9 ай бұрын
If most of these are e-commerce only
@MARTIN-101 8 ай бұрын
what is the name of your company ? are you on linkedin ?
@shannoncole6425 3 ай бұрын
Wouldn’t it be better to build tools that solves specific problems more than trying to convince people they need tools they don’t know about? For example I pay more than I want in my business to access a tool that does a small part of a problem I’m trying to fix. I found another tool that fixes the whole problem and I’m happy to spend $30 a month to solve that problem.
@nyosdroid868 11 ай бұрын
I did similar projects in the past (in php long time ago with curl and tidy, now in python with bs4 or selenium). To avoid bot detection, you can use the tor network, detect captcha request, and simply reconnect tor to get a different IP. Of course, you still need to put random sleeps here and there. Also, you need to be prepared to update or rewrite your code any time the website changes. Nice video!
@sergesmitty137 11 ай бұрын
"Gathering" data from other people is one of the holy grails of programming. OpenAI is the epitome of this aren't they - Gathered data from all of the internet, regurgitated it with "AI magic" and sell it back to consumers as their own.
@DerickMasai 11 ай бұрын
@@sergesmitty137 You say that in quotes as if it is some gimmick yet multiple multi-billion dollar corporations weren't trying pouring a fuckton of money into similar projects and all failed or produced AI barely worth mentioning. Besides, they aren't selling the data they scrapped, that was just used to train the model, they're selling you a tool that can do practically anything digital you'd hire a minimum wage employee to do, for now.
@DerickMasai 11 ай бұрын
As for the original comment, thank you so much for the Tor idea! Never even thought using that as the useragent. Downloading it now.
@SaKa-L 11 ай бұрын
What an idea , using Tor networking for web scaraping . Thumbs up for that 👍
@nyosdroid868 11 ай бұрын
@@DerickMasai The UserAgent string is simply an up-to-date Firefox/Chrome. What I meant was to use tor as a proxy to hide and replace your real IP quickly. tags=b.findAll('p',attrs={"class":"mt-4"}) # whatever style their captcha text is using.. if(len(tags)>0 and 'Your IP made too many requests' in tags[0].text): # we got caught print("CAPTCHA, restart tor") restart_tor() and that function does: def restart_tor(): try: print("restarting tor..") time.sleep(random.randint(10,15)) x=subprocess.run("sudo /etc/init.d/tor restart", shell=True, timeout=30) except: return False print("done, waiting") time.sleep(random.randint(25,30)) return (x.returncode==0) I hope the indents stay ok. Anyway: just detect if the target site caught you scraping, and instead of solving the captcha, just restart tor. When it reconnects, the new IP you get can redo the last request. (Also I've put my user into the sudoers so the script can do the restart part without my password.) The timing magic numbers were a bit more sophisticated, but it works.
@codesandtags 10 ай бұрын
I totally agree, I've done a couple of web scraping scripts for some companies, and to get some specific data, however there are a lot of challenges. But in general the rate limiting for APIs and changes in DOM, or catchaps are the most common ones.
@normallyChallenged 10 ай бұрын
What does DOM stand for?
@74G_ 10 ай бұрын
@@normallyChallengeddocument object model
@camilocastrillon2030 7 ай бұрын
Since react doesn’t modify the dom directly I’m guessing I could take an advantage and sort that out, right?
@Dom-zy1qy 5 ай бұрын
@@camilocastrillon2030react does modify the DOM. The dom is updated with the diff of the react virtual Dom. This isn't really the issue with web scraping & Dom changes though. They mean "changes to the DOM" as in the websites html just changes as the developers of the website add new features, change the layout, etc.
@Brodragon2225 4 ай бұрын
@@camilocastrillon2030 now its saturated hey can u think of how free saas app can make money
@LeoTang2307 10 ай бұрын
I'm trying to build as many projects as I can right now to upgrade my skills and cv. This project idea is really helpful to my resume, I'll try to do it using fastapi and mysql (if possible I'll try to track stock price). Truly appreciated for sharing this project, sir!
@Soulseeologia 10 ай бұрын
How did it go
@SpacefootLife 11 ай бұрын
Great tool Tim, i look forward to playing around with this. I still have a lot to learn about the Data industry though, keep up the amazing work.
@TechWithTim 11 ай бұрын
Glad it was helpful!
@nitindevatraj 11 ай бұрын
Need a whole series with videos like these
@yogeshnarayankar951 11 ай бұрын
+1
@EddyWydraTV 11 ай бұрын
Agree
@uboxtech 11 ай бұрын
+1
@spicer41282 11 ай бұрын
3rd to 6th the MOTION & Request!! Thank you in advance!
@egedesolomon7454 7 ай бұрын
Agree
@Autoscraping 4 ай бұрын
A wonderful video that we've used as a reference for our recent additions. Your sharing is highly appreciated!
@flowbotgirl 10 ай бұрын
Thank you! This is the replacement for the outwit browser I've been searching for!
@0x656e 11 ай бұрын
I love web scrapping; web scrapping is really satisfying when you finish your project.
@dariocannistra3322 2 ай бұрын
Hey Tim, you always have amazing content. Keep it up! Greetings from Italy!
@krvns 11 ай бұрын
Thumbs up if sponsored tools are not your content!
@thetruthsayer8347 11 ай бұрын
But man’s gotta eat tho
@user-nq2jx6md4h 10 ай бұрын
@@thetruthsayer8347That’s his 1 million Dollar project (Let viewers use the sponsor) 😂
@johnsamuel6080 10 ай бұрын
His bills won't sort themselves out
@11bftw 10 ай бұрын
He mentioned that companies are fighting against scraping...
@adamrhea2339 10 ай бұрын
He literally used the sponsor's tool in the project. I would use the tool regardless if it was a sponsorship or not so in this case case he makes money at my benefit. I have no problem with this.
@Faisal1504 6 ай бұрын
Very helpful looking for data will check out bright. Thank you.
@enlyxtyrill6127 10 ай бұрын
Wow! I was literally thinking of doing the same thing, but really had no idea it was that hard or that some many people have already tried the same thing and failed. Before I even started facing the problems, you gave me the solutions for free... So thank you!
@philphillips907 10 ай бұрын
Tim, your videos look very professional. I see the gear/hardware you use. May I ask what software you use to record your screen and include a picture-in-picture of yourself? I did not see this info anywhere. Thanks in advance!
@kamilmarszakowski8694 10 ай бұрын
That's awesome, keep going bro
@SamuXZamorionVII 11 ай бұрын
He is right, I work for tech consulting company. One of the client projects I work on is web scraping project that collects doctor disciplinary action data using python and they pay a lot for us to do that for them.
@anonfourtyfive 11 ай бұрын
dude, I need money. I'm literally scraping day and night. is there anyway I can help ? I'm hungry for data.
@vatsalyavigyaverma5494 11 ай бұрын
@@anonfourtyfivehow you store those data 😊
@anonfourtyfive 11 ай бұрын
@@vatsalyavigyaverma5494 there's multiple way of storing data... I prefer Tinydb or SQlite, but if you work on a bigger project, I would recommend using a cloud storage like mongodb or firebase.
@pawsstory1288 6 ай бұрын
@@anonfourtyfivewhat is your goa scraping day and night?
@alimihakeem841 24 күн бұрын
Thanks for this valuable content Tim. I found it helpful
@Coole986 10 ай бұрын
20$ / GB is huge and you will hit that really fast lol
@LalitaPaswan-yq6ll 11 ай бұрын
Sir i watch your all videos and they are very helpful thankyou to provide as like that informative content.....👌
@shitinsideyou 10 ай бұрын
Bright Data makes it easier to steal data from websites that try to protect themselves... Great job BrightData!
@simplyeleza 8 ай бұрын
thanks for the video. I have been thinking of making something similar. Even bought a domain for it.
@mad-hat 9 ай бұрын
Thanks for inspiring me to learn programming, you're really cool. This is a very cool and interesting project. ⌨💪🧑💻
@miyu545 Ай бұрын
A simple Chrome extension build would have made this so much easier for you and it's completely transparent for bot blocking.
@viktorvegh7842 8 ай бұрын
Great stuff Bro i was wondering how to do this for long time
@RolandoLopezNieto 5 ай бұрын
Super helpful, thanks
@tylerk2913 11 ай бұрын
Great video ❤ one thing I noticed that might be helpful on your code is comments when you’re walking through what each part does. That way anyone who uses it from your GitHub doesn’t have to reference your video each time to hear what it does. Just a small detail I noticed. Otherwise, fantastic idea and project!
@ttpoh4688 10 ай бұрын
ChatGPT will do that for you.
@Entertainmentexe 11 ай бұрын
Please keep project ideas video coming 🙏
@guidoumfaroukomar6282 11 ай бұрын
Man u gave us so many valuable informations,we appreciate it❤
@wmelon3392 11 ай бұрын
thank you my king this sounds so cool
@workin3d 5 ай бұрын
Good idea. If u r in 2013. That's plenty of services who are provided same possibilities
@derrickpham7987 10 ай бұрын
Thanks for info
@Militarychannel12 11 ай бұрын
Hey i watched some of your videos and i like what you do here ... i was thinking may be if you could do a tutorial on how to Develop an API with Python or JavaScript
@Virtual869 11 ай бұрын
very great vedio like every time
@brianhauk8136 10 ай бұрын
How many pages can you scrape using this approach before Amazon blocks your scraper, and what's the best approach to avoiding detection by Amazon and other sites?
@HappyAnimals3D 10 ай бұрын
Thanks Tim. Any plan for a detailed tutorial?
@awakenwithoutcoffee 2 ай бұрын
appreciated brother. I was actually building something very similar but yours is definitely better build. Do you recommend against doing this in Javascript ?
@screweddevelopment12 11 ай бұрын
I did something like this for a company I worked for. Setting up slowly changing dimensions tables in python is p hard when there aren’t primary keys for the data you’ve scraped.
@TannerBarcelos 11 ай бұрын
This is the issue I am in right now. I am scraping tons of data but I need to create the data model but I am lacking primary keys, etc.
@LGL1999 9 ай бұрын
@@TannerBarcelos Hey, did you figure it out?
@planktonfun1 11 ай бұрын
I did this back then with food delivery services for example burger it will display the cheapest delivery and price from a pool of available service.
@isaacbawangisah6096 11 ай бұрын
Tim deserves more than 1.26 million subscribers. Fantastic job!
@ordep.dutra1454 11 ай бұрын
Thanks for sharing this project with us !!! 👍
@subhranshudas8862 11 ай бұрын
This product would be a smash hit among Coupon Grandmas in Texas.
@pauljoseph3081 10 ай бұрын
I've seen people exploiting online job portal sites like Upwork. They hire freelancers from the Philippines to scrap websites rather than building a program, and of course they *CHEAP AF* like we're talking about $2-3 per 100+ websites. Most of them charged $2-3 per hour which is absolutely disgusting.
@user-mz8rz9lk8n 11 ай бұрын
What language would be the best to learn to create an app? Java? C#? Which one should I choose? Also, I want to know the prospects of each language. Can you tell me?
@felixinit 11 ай бұрын
Amazing project scraping the web
@rishiraj2548 11 ай бұрын
Thanks
@bengabp 11 ай бұрын
I am planning on working on something like this, but I am from africa and i dont have too much knowledge of these GDPR laws and so on, could you make a video on that please?
@KrAsHeDD 11 ай бұрын
Thank you
@Alufakwa 11 ай бұрын
Great project
@heliahaghighi3597 11 ай бұрын
Hey, thanks for the great content! Guide me please, what is the road map for learning the tools to understand what you have done in this project and be able to do it on my own?
@MARTIN-101 8 ай бұрын
you can think of a project of your own. and try to make it. if you ran into problems than search on internet. but these kinds of projects are tricky. so you need solid advice. and one thing you do will not work after few months. 😂
@MARTIN-101 8 ай бұрын
one time i found a scraper on internet. but it was slow. so i made it 50x faster. you can think of it like this too. find someting and than optimize it.
@PeriklesPeriklesoglu 6 ай бұрын
thank you
@pythoneatssquirrel 8 ай бұрын
The idea is cool(the frontend part) except the "pay us to run your code and provide you the data data your code got" part, but what if you would want to scrape like 100k distinct items? The frontend will become just a long scrollbar.
@pythoneatssquirrel 7 ай бұрын
I build a "free" scraper to this project that you can find in my videos.
@mgusa9372 10 ай бұрын
Tech With Tim, dude, big fan, suggestion, get a ton of sleep like a TON like 24 hours then eat and take a shower then a really cold shower, and then go get a professional picture made or just use your camera and then use that pic as your profile pic cause on your profile pic u look like you're going to pass out from too much hard work too many hours (which I understand being a techie myself) your eyes are giving you away. Love your stuff man, thank you tons, keep it up PLEASE, TKX!
@cuturbs 11 ай бұрын
I tought I had some good skills on web scraping, then you showed me I'm just a baby still. Loved the idea and the fact you gave it out for free
@charrow10 10 ай бұрын
not free, it is just an ad
@MARKOTHEDEV 11 ай бұрын
I love this dude deeply this was the reason I started learning code❤ now some one is give me a repo
@iocyber 11 ай бұрын
this is so sickk ima mess with it rn
@JTJaniceTek 6 ай бұрын
hello Tim, is it possible to make a video on how to download the newest version of chrome driver, been stucked on it for days... i wanna crai
@coddude3284 11 ай бұрын
Mr Tim is automation with java a a good thing ? wich better java or python for automation
@susanwojcickiisalonelyrath7311 10 ай бұрын
This is exactly what I was looking for - thank you!
@iluvmybruddas3072 10 ай бұрын
bright data ok cool got it !
@cmd_processor9168 2 ай бұрын
Thanks for this video. How did you bypass Playwright default timeout? It sees that Playwright has a default timeout of 3000ms regardless of setting await in line 106 to 60000 or even Wait=commit
@theDemolisher13 2 ай бұрын
Just wondering haven't played with the program yet. But does the program also record prices differences if items have multiple versions example Red rain coat vs the same rain coat but it's Blue and cost $1 more.
@reginmaru7938 11 ай бұрын
Legend!
@zeburgerkang 6 ай бұрын
I'm trying to scrap documents from a website with multi layers, so I can use those files with a logical chat bot to reference what I'm looking for, please any help will be appreciated, thanks in advance.
@maksimluzin1121 11 ай бұрын
Hm-m-m... Interesting approach... I'll definitely will learn it later, since I've used only the simple Web Scraping solutions, and it doesn't work out sometimes... Finger up, Tim, for your work, anyway!!!
@demaxl732 11 ай бұрын
I'm definitely adding this one to my resume
@pedromonteiro8792 10 ай бұрын
Isn't Web Scrapping Illegal? Some time ago I had the idea to do a project just like this one and I ended up giving it up because I thought I was illegal. I read it on the website's terms and conditions.
@AliAliOxenFree 11 ай бұрын
Great video. scrapers + llm apps are going to dominate data very soon (if they aren't already doing this)
@tanysanchez8519 11 ай бұрын
What are llm?
@sbussisodube1084 10 ай бұрын
@@tanysanchez8519 Large Language Models (machine learning algorithms)
@jeremyyd1258 11 ай бұрын
More uses: - job searches - combine with NLP to obtain live information regarding sentiment towards products or services
@coder_117 11 ай бұрын
What's NLP?
@TauvicRitter 11 ай бұрын
@@coder_117natural language processing. Analyses text
@satwikawasthi2002 9 ай бұрын
Can we run whole project without brightdata functionality for website which do not blocks or medium level security in it?
@3x10.8_ms 11 ай бұрын
youtube videos these days are becoming ads
@superlivescore001 10 ай бұрын
Hello Tim, i'v got problem mostly with style. Css even with the guide of chat gpt, advise me please
@rafaelfernandez742 10 ай бұрын
Who is buying scraped data - companies would just build it ? or is the value proposition any community that wants/needs to know if data has changed and building some kind of tool to do that?
@lettalkaboutit 10 ай бұрын
Hi, can you make a video on how to deploy your web scrapper app on a online VPS please?
@proxyscrape 6 ай бұрын
Don't forget the proxies while web-scraping 😄
@emmanuelrobles5386 11 ай бұрын
Would you offer a scaper in RapidApi or any other marketplace?
@killua7270 11 ай бұрын
Can you tell, what thing did you do to monetize through webscraping
@mandongpedroza7486 2 ай бұрын
good day tim, i just want to mention and ask permission that i will extend this base code. for now i made a private repo and soon on the right time i will make it public and will credit you. thanks
@Alldeveloper2356 4 ай бұрын
Do you have a video of web scraping data using javascript and nodejs?
@yashinshekh802 9 ай бұрын
instead of using any third party library can we integrate scrapy in it
@papasedokarajeh6183 10 ай бұрын
I want to create a search algorithm that can find files that contains a particular word or sentence and open it using python. I save passwords and other things to remind me using note pad but I want a way of typing a word or sentence or part of a sentence and have the document open and the word highlighted.
@ryanhoughton7415 7 ай бұрын
I can build a web scraper in Python selenium that spits data into a csv. But where could I go to learn to do something like this?
@akeipra Ай бұрын
Does anyone has a good tracking tool for specific website or search enginges for example if any tender is realsed?
@aminekoumni5075 9 ай бұрын
In order to run it every day for example you should host it an make a cron job for it ?
@Lifestrengths 11 ай бұрын
I'd like you to build a Encryption and decryption tool
@X_x_kingfisher_x_X 11 ай бұрын
Hey Tim, i've learned some basic python from youtube, practiced in codewars, currently approaching 1kyu, i really love coding, i want to take it seriously, i wanna be job ready, i've no guide or mentor, i am looking for some serious advice and guidelines, because just feeling like a radarless boat, i don’t know what to do next.....,please help, and can i help you anyway?
@khnykaq8823 11 ай бұрын
Hi Tim. Thank you for your videos. Recently I watched your video about Selenium and your site in the video was very slow. Is it because of the slow server or because of Python? I've read a couple of articles about web developing stacks today and Python was mentioned just once. Does it mean that for a web developing it's better to you somthing else except Python?
@coddude3284 11 ай бұрын
java is so much faster
@learner8053 11 ай бұрын
Can you please build a live streaming web app & show us?
@7DuRd3n 10 ай бұрын
that's great
@yamani3882 6 ай бұрын
Just the cost of Bright Data makes this not applicable. I want to make 10,000 requests a day at least. Anyone found a work around?
@thenextcr7 10 ай бұрын
this looks like django? can you talk about deploying it? idk why i had such trouble with it
@themindlesstruffle 11 ай бұрын
Would you consider a full length setup and code tutorial for this project or a similar one?
@stanbloem6316 7 ай бұрын
!
@quinndaisies6949 11 ай бұрын
Funny enough I used PyQt5 to build a web scraper, I can scrape Amazon without getting block... But with this new information I can extend my web scraper a lot further to make a more complex project. Thank You Very Much Tim...
@mohammedmagdy3058 9 ай бұрын
can i change it to amazon eg?
@vexymous 11 ай бұрын
I think with ChatGPT interpreter, a lot of possibilities are opened
@kariakichernocherno2274 11 ай бұрын
we watch you grow up
@TopStrikerMaverick 2 ай бұрын
Can u use no code like bubble for this
@Grinwa 11 ай бұрын
The biggest question is where to sell how to build trust I already made few tools and selling them but i have no idea how to sell web scraping data or tools Also thanks ❤ that's a good starting point Edit: 9:02 after seeing this i remember i made something similar but it scraps anime each one with it episodes and each episode with stream url and title desc... and then display and stream anime with electron ui but i never finished it cuz i have no idea how to make ui with js
@Magicallstore 10 ай бұрын
can you do a scraper for movies?
Қайрат Әділгерей, Muyun Brothers, Аманғали Сыпабеков, Нұргүл Даубаева ҚЫЗЫҚ TIMES | Хотя бы кинода 3
Marat Oralgazin
Рет қаралды 611 М.