Meet ChangeDetection - A Self-Hosted Website Change Detector!

Рет қаралды 23,799

Күн бұрын

Thanks to Docker for Sponsoring this Video! Get Actionable insights for software supply chain management with Docker Scout! dockr.ly/45HWDnR
Tracking things on the web just got a whole lot easier with ChangeDetection, the free and open source Docker container! Track website changes, price changes of products, and even track out of stock products with notifications all from a container you host yourself!
Video Notes: technotim.live/posts/change-d...
ChangeDetection on GitHub: github.com/dgtlmoon/changedet...
Support me on Patreon: / technotim
Sponsor me on GitHub: github.com/sponsors/timothyst...
Subscribe on Twitch: / technotim
Become a KZfaq member: / @technotim
Merch Shop 🛍️: l.technotim.live/shop
Gear Recommendations: l.technotim.live/gear
Get Help in Our Discord Community: l.technotim.live/discord
2nd channel: / @technotimtinkers
(Affiliate links may be included in this description. I may receive a small commission at no cost to you.)
00:00 - What is ChangeDetection
02:33 - Requirements
02:59 - Configuring System
03:56 - Docker Compose Explanation
05:24 - Moving Config to Docker Machine
06:40 - Starting the Container
07:24 - Exploring ChangeDetection Dashboard
09:34 - ChangeDetection Job Settings
11:21 - Global Settings
11:58 - Be a Good Bot
12:18 - Global Settings Continued
14:15 - Be a Good Bot Part 2
15:10 - Adding a New Site
16:29 - Headless Browsers and Drivers
17:59 - Enabling Chrome Browser and Playwright
21:17 - Using Chrome to Render Pages
23:47 - Using Chrome and Playwright to Monitor Deeplinked Sites
28:55 - Price Tracking
30:33 - Restock Tracking
33:15 - ChangeDetection Docker Container is Powerful!

Пікірлер: 67

@owen2507 10 күн бұрын

A big use case for this (at least for me) is scholarship/grant websites. They tend to rarely be updated until they become available for the current scholarship round. For example, there is one scholarship I am looking at that still has the "2023 deadline passed" banner, and I am waiting for it to be updated for 2024. This will be incredibly useful. Thanks for the video!

@TechnoTim 10 күн бұрын

Great idea!

@TheOisannNetwork 9 күн бұрын

I've used this for years. It's an incredible tool and a real must-have.

@fanaticdavid 10 күн бұрын

I recently read a forum post from someone who was using a Docker tool like this one to track the availability of 3D printing filament, and I meant to ask them which tool they were using or do some research myself. But here you are, delivering exactly what I was looking for, as if you have a sixth sense for these things 😂 Thanks, Tim!

@TechnoTim 10 күн бұрын

Sounds like this will work great!

@WamocIsolda 10 күн бұрын

I plan on using this to monitor the county website for people trying to steal my property with a forged deed so I can fight it and fix it quickly.

@diegoalejandrosarmientomun303 6 күн бұрын

Dang, this is huge. I was looking for this for a while ago now. Thank!

@ExpressITTechTips 10 күн бұрын

Now this is a tool I can use great find ❤

@joelfrojmowicz 10 күн бұрын

Great video. Thanks Tim!!

@kevinhughes9801 9 күн бұрын

Wat a great bit of software with so many uses. Thank you for great breakdown

@theSimonManuel 5 күн бұрын

I tried to configure Huginn about 2 years when I first started using Linux. Did not understand head or tail. Came across ChangeDetection a little over a month ago. Most practical piece of software I host.

@nezu_cc 5 күн бұрын

Huginn is really good if you need something very specific or you need to do a bunch of post-processing on the data. Huginn helped me buy a Raspberry Pi during the supply shortage

@DavidDavisL 9 күн бұрын

This is a great tool and I have used it quite a bit in the past for specific purposes. As mentioned - be sure to avoid an overly aggressive scanning interval, especially if running this from home. Many sites will block suspected bot scanners - and that is a very bad thing when your home IP is what is being blocked! Running from a cloud instance or through a proxy should at least avoid the problem of getting blocked at home, but it is best to avoid the situation altogether.

@TechnoTim 9 күн бұрын

Well said!

@lloydsshednanigans 9 күн бұрын

Thanks for the video. Wonderful idea. I'm playing with it. It's quite the touchy little creature isn't it?! It's favourite thing to say is "Exception: Target page, context or browser has been closed". Still trying to track this down but it's an interesting journey.

@GitPush1 9 күн бұрын

I'm having the same issue on an Unraid server. Still trying to track it down

@lloydsshednanigans 9 күн бұрын

@@GitPush1 I've been playing further and it seemed to be much worse when I used Tim's github link that he showed for a test. Perhaps loads of people are using it in the example and Github is getting annoyed? Just a thought. After taking that test out, it seems to have settled down a bit now. Are you using the example too? It may not be related but it was just a thought.

@pax0707 8 күн бұрын

Might have missed it but "Visual Filter Selector" is a great option too. Think it shows up after the 1st screenshot grab.

@ImARichard 10 күн бұрын

Good ol XPATH queries. Id be curious how often the elgato recorded steps get broken by Elgato redeploying their site. But it is nice that they give you the option to customize it to be more resilient. Awesome awesome video! And a fantastic tool that I dont know what Ill use it for, but definitely going to add it to my arsenal.

@TechnoTim 10 күн бұрын

Thanks! I thought my xpath days were over 😅

@AtindriyaDas 9 күн бұрын

Great! Thanks very much!

@basheerparvez 4 күн бұрын

Great video. Can I ask what terminal are you using in the video?

@narutokunn 9 күн бұрын

Great video. Thanks TT

@dustinrouillard 10 күн бұрын

Level1Techs spotted 👀

@TechnoTim 10 күн бұрын

If Wendell is Level1, I am more like Level 0.01

@faithful451 10 күн бұрын

Really cool, thanks

@AndreaGhensi 9 күн бұрын

I'm curious to know why you don't use docker volumes for the datastore, since you don't have to access/ edit those files... to me it feels that, in this particular case, bind mounts are just a needless complication (and going with volumes will unlock gitops and remove the need for ssh, but I get that it's an advanced topic). That said, I will definitely try this service!

@TechnoTim 9 күн бұрын

Thanks! Unfortunately this is also where your sites are stored as well as your dashboard config. Agreed you can use a Docker volume but I've found bind mounting it to the file system when using Docker (vs. k8s) is much easier to manage, back up, and not accidentally delete 😅

@mjl4878 9 күн бұрын

Great video, biggest issue I'm seeing is it takes forever to check. Running on a proxmox docker lxc container with 8gb of ram. Noticing this only after adding Playwright

@Saintel 9 күн бұрын

Love this video. Anyway you can do more software installs for those that use Synology Nas systems?

@TechnoTim 9 күн бұрын

Thank you! What are you looking for specifically for Synology? I don't have one but assume they just use a web form to fill out?

@alexandzors 10 күн бұрын

The timing of this video and me seeing a comment in an unrelated Discord server about this service.

@userou-ig1ze Күн бұрын

Is there an LLM integration? What we really want is a vision model like llava2 to navigate the steps, not xpath

@iamrage4753 10 күн бұрын

Thats really good software

@ubeaut 9 күн бұрын

I use puppeteer to get my water consumption from the council website every day as they have installed wireless water meters which send the consumption to the council. Similar to what you are doing.

@TechnoTim 9 күн бұрын

I love puppeteer. Use it all the time in CI.

@kriffos 7 күн бұрын

Playwright is basically a fork of puppeteer … the API is mostly the same.

@mattb474 9 күн бұрын

Just ran this up myself, noticed you always cut your video when you do a fresh check - is that because you came across this error "Exception: Target page, context or browser has been closed" and pretty consistently needs to be run a couple of times to get a clean check. Or were your checks all clean and just needed a few moments to run?

@TechnoTim 9 күн бұрын

I usually cut out long pauses of the video. I did run into that initially when using the web driver + selenium but not since moving to playwright and chrome. You may need to adjust some of the settings but if it only happens on the first initial runs it might be hard to track down unless you look at the logs when it happens.

@MacJFitness 10 күн бұрын

Is there any ethical way to stop Amazon captchas? Even when setting checks to every 1 hour I am still hit with a captcha.

@TechnoTim 10 күн бұрын

Amazon has the best bot detection on the planet... seriously

@JZL003 10 күн бұрын

Keepa or camel camel camel?

@DuckSport 9 күн бұрын

Those explicit xpaths -shudders- Is it possible to customise the selectors so you don’t end up with horribly flaky browser steps?

@nezu_cc 5 күн бұрын

yes, that's the reason why it waits for you to click apply so that you can customize the step

@DuckSport 4 күн бұрын

Nice, thank you. Is on my kid to try out :)

@kalam564 10 күн бұрын

I've been self hosting this for a few months now to monitor Vanguard 7 day SEC yield on VMFXX. Great piece of software.

@TechnoTim 10 күн бұрын

Awesome use case!

@squalazzo 9 күн бұрын

what about sites that use MFA?

@TechnoTim 9 күн бұрын

I think you might be out of luck with that unless you can find a way to automate getting that code and supplying it, unless you only need to supply it once. If that's the case, you can drive the browser to supply it once and then hopefully the cookies are preserved so that the next time it runs it uses those. If not see if there's a way to persist those cookies between runs.

@squalazzo 9 күн бұрын

@@TechnoTim I was thinking the way alexa media player integration works in home assistant, it register as a 2fa client... But that means if someone breaks into your ha, has your full Amazon account