Server Monitoring // Prometheus and Grafana Tutorial

Рет қаралды 610,174

Күн бұрын

Server Monitoring with Prometheus and Grafana setup in Docker and Portainer. I explain the difference between metrics and logging and how Prometheus can monitor all your server metrics and use Grafana to visualize them.
Teleport-*: goteleport.com/thedigitallife
Related Videos/Links
• My new Proxmox Monitor...
________________
💜 Support me and become a Fan!
→ christianlempa.de/patreon
💬 Join our Community!
→ christianlempa.de/discord
________________
Read my Tech Documentation
christianlempa.de/docs
My Gear and Equipment-*
christianlempa.de/kit
________________
Timestamps:
00:00 - Introduction
01:08 - Why centralize monitoring
02:01 - Difference between logs and metrics
03:19 - What is Prometheus?
03:46 - Monitoring Architecture
05:12 - Deploy Prometheus and Grafana
10:03 - Configure Prometheus
13:21 - Third-Party Exporters
19:04 - Visualize data with Grafana
21:44 - Import Grafana Dashboards
________________
All links with "*" are and/or include affiliate links.
#Prometheus #Grafana #HomeLab

Пікірлер: 303

@christianlempa 2 жыл бұрын

Please be aware: update to the Git Repo: github.com/xcad2k/boilerplates I now separated the docker-compose files for Grafana and Prometheus in two separate folders!

@giftcp82 2 жыл бұрын

why. is there any advantage to doing this or were there challenges with the first configuration.

@Trains-With-Shane 2 жыл бұрын

@@giftcp82 In case you wanted to set them up standalone. Not a problem for somebody who's familiar with docker compose YAML files but for somebody just starting out it could get confusing having it all together. This way it narrows the focus and doesn't get everything potentially confused for the new user.

@AbdulRahmanKayaliIbnMukhless 2 жыл бұрын

What is the difference between grafana/grafana and granfana/grafana-oss as your updated Git Repo refers to the latter while video refers to the former?

@mridulranjan1069 2 жыл бұрын

With time any updates like this will be harder to find in the Comments section. I was lucky I found just when I was wondering the video shows a combined docker-compose.yaml but the actual repo has them separated. Maybe update the main Description section with a dated update about the repo? You know, just like a code description update :-) Anyways, loved your video so far, the presentation. Great Job!

@mehammered Жыл бұрын

The exporters folder seems to be missing also. Edit: Its in the dir is the prometheus folder not within it.

@cyberlord64 2 жыл бұрын

Absolutely invaluable. I have gone through a few of your other videos as well, and everything is essentially easily digestible essential knowledge that would otherwise take me weeks to search and figure out on my own. A huge plus is that the scripts and snippets provided actually work out of the box and everything is explained on an almost line by line basis. Thank you for your amazing work

@adeelhashmi145 Жыл бұрын

wait what? invaluable?

@bradleydiggs Жыл бұрын

@@adeelhashmi145 It's weird that it means what it does (with the prefix in- that normally means the opposite), but it actually means "incredibly valuable" or "more valuable than valuable"

@user-se5xg3vz8p Жыл бұрын

@@adeelhashmi145 priceless

@xx482 2 жыл бұрын

Before going to watch this video .. i really appreciate you for making this video .. i was struggling to learn the basics of Prometheus and grafana , you really helped me by putting this video.. Thx in advance

@christianlempa 2 жыл бұрын

You're welcome!

@wildflowers465 Жыл бұрын

Hey Christian, another amazing video! I set this up in my home lab, and it works great.

@Psyt0s 2 жыл бұрын

Man, I can't thank you enough!! all your videos are so easy to follow and actually understand what I'm doing!! You are awesome!!

@christianlempa 2 жыл бұрын

Thank you so much :)

@Flako-dd 2 жыл бұрын

Awesome man, a youtube guide that doesn't start with notepad on windows, but immediately gives you the github repo. Love it, great guide!

@christianlempa 2 жыл бұрын

Thank you so much! :)

@tirushv9681 2 жыл бұрын

Happy to see this video particularly by "The Digital Life" Thanks sir cause i have seen many videos already but here we can get exact points to remember and its basically on point

@christianlempa 2 жыл бұрын

So nice of you!

@neuralnetdev Жыл бұрын

for anyone who cannot get the prometheus or grafana container to start or they keep restarting (which is what happened to me), use the details below: the volumes that your container accesses, run this command on them: sudo chown -R 65534:65534 prometheus_example/ replace "prometheus_example" with the folder name that you provided to your volume(s) 65534 is the user and group id of the user/group that runs prometheus in the container. for me, grafana also had permission issues in the logs of the container, too. the user and group id is 472 so you'd replace "65534" with "472" in the command above. this issue occurs due to the user inside each container not having the correct permissions set. to my knowledge, most people here shouldn't have a problem if you followed the steps provided in the video correctly (which i didn't and i changed a few things with the volumes which is probably why i had this issue)

@processwire-rocks Жыл бұрын

Great video and super helpful as always! Thx for all your great content!

@itx777 4 ай бұрын

Thanks Christian for all your videos, they are very informative!

@renobodyrenobody Жыл бұрын

Thanks a lot. There is so much ressources, sometimes ununderstandable, it's difficult to find the information hidden in the noise. Your video is very very helpful. Thanks a lot!

@0917hubing Жыл бұрын

今天边学习，边自己服务器配置了监视系统，谢谢Christian Lempa。

@VincentBlouin Жыл бұрын

You've been an essential piece of the puzzle in setting this up.

@RichardExists 2 жыл бұрын

Nice walkthrough man, very robust! Funny, I just did a small lab about these last night! So awesome!

@christianlempa 2 жыл бұрын

Cool! That's great to hear, thank you btw 😄

@huplim 2 жыл бұрын

Just recently stumbled upon your channel and just needed to tell you your videos are awesome. Subscribed!

@christianlempa 2 жыл бұрын

Thank you so much 😃

@kalingsrajan4017 2 жыл бұрын

Good explanation and got some new in this tut. Thanks for your great work.

@imanshirkhodaee1704 8 ай бұрын

All 24 minutes was useful and informative

@samme4life 8 ай бұрын

Quite informational video. Thank you for sharing your knowledge

@tyu3456 2 жыл бұрын

Amazing. Thanks so much for this excellent walkthrough

@christianlempa 2 жыл бұрын

Thank you! :)

@Bradenxd12 2 жыл бұрын

Such a great video! I've been developing a Grafana plugin at work and there is large void in high quality Grafana content.

@christianlempa 2 жыл бұрын

Thank you so much! :)

@IgorKuts 7 ай бұрын

Thank you, Christian, for this ultra-high quality content. Learned a lot. One thing tho, as for me, frequently i found it difficult to keep up with your pace.

@JacquesvanWyk Жыл бұрын

great video. I have been looking for months for a solution like this.

@christianlempa Жыл бұрын

Thanks!

@md.mainuddin8211 4 ай бұрын

Thanks a lot for sharing with us

@JCtheMusicMan_ 2 жыл бұрын

Get hit with a firehose of information! I have found it difficult to find quality tutorials on setting up a Grafana/Prometheus dashboard. Thanks for the wealth of resources and information!

@R_Mahesh5 Жыл бұрын

Very useful, thanks a ton!

@leboncoinyca4704 7 ай бұрын

Great video as usual. Thanks !

@christianlempa 7 ай бұрын

Thank you :)

@cezwitsteez 2 жыл бұрын

thanks for this tutorial man, super helpful!

@christianlempa 2 жыл бұрын

Glad you liked it!

@gaifut Ай бұрын

Thank you for this detailed explanation.

@christianlempa Ай бұрын

Glad it was helpful!

@Dravig Жыл бұрын

Great job! Thank you!

@George-zm4iu 2 жыл бұрын

How have i only just seen your channel now! Awesome video, i enjoyed your ad too lol. Sub gained :)

@christianlempa 2 жыл бұрын

Awesome, nice to hear you're liking the content 😁

@hareeshab9752 Жыл бұрын

Great video, it help us to understand very easily. Nice work. Always waiting for your new videos to upgrade ourself

@christianlempa Жыл бұрын

Thank you so much 😊

@hikmatullah9463 6 ай бұрын

It's was a nice video about Prometheus and grafana

@TheLichKng64 4 ай бұрын

thank you so much! your videos and content are super useful!

@christianlempa 4 ай бұрын

Glad you like them!

@jacobcrisman6834 Жыл бұрын

I don't know why I try to go anywhere else when I need something installed in Docker. These videos always give more information than what is needed and everything is explained very well

@christianlempa Жыл бұрын

Thank you! :)

@locosporasturias 2 жыл бұрын

Amazing, thank you very much for your time

@zb3485 Жыл бұрын

the channel is incredible, thanks!

@christianlempa Жыл бұрын

Thanks! You're welcome ;)

@benjaminshtark5977 Жыл бұрын

simply amazing stuff!! I really like 2 things about your videos 1. amazing and professional explanations 2. your German accent :DD

@christianlempa Жыл бұрын

Thank you so much ☺️

@averagedev7768 2 жыл бұрын

Used cockpit for the last 2 years. I am very happy how it works. I have 7 servers in total i have to manage, and cockpit allows me to do just that by logging in on one single server adding others to there and watching there progress

@christianlempa 2 жыл бұрын

Nice, yeah cockpit is pretty cool, I also use it sometimes ;)

@martinlortie 2 жыл бұрын

Yep after turning on Cockpit this is exactly what I needed, but if you have more that one server with multiple instances of things then I can see Prometheus/Granafa shine.

@Lucas-gp6xn 2 жыл бұрын

Beautiful Tutorial, thanks!!!!

@christianlempa 2 жыл бұрын

Glad you liked it!

@chrisumali9841 2 жыл бұрын

Thanks for the demo and info, have a great day

@christianlempa 2 жыл бұрын

You're welcome! :)

@chrisumali9841 2 жыл бұрын

@@christianlempa which extension did you use for visual studio code to enable ssh editing on the server? Remote ssh by Microsoft? Thank you

@sammanakkuon8581 7 ай бұрын

Thank you for sharing

@DigitEgal 2 жыл бұрын

Wow, there is so much information in this Video i have to look that when im not tired :D But i can say already: Great Video mate! Also i really like the "on camera writing" presentation uve done at 3:46-5:00 since i also learned a lot stuff from Networkchuck when he was doing that. Atleast of course i can also see the small improvements (round corners) youve done at the cam-picture implementation and improvements in structure of your presentation.

@christianlempa 2 жыл бұрын

Thank you so much! Great to hear that all the small tweaks help you to get a better experience 😁

@ulrichbeutenmuller8101 2 жыл бұрын

Awesome. Will put this on my test lab asap.

@christianlempa 2 жыл бұрын

Thanks mate that sounds great!

@ryanbell85 Жыл бұрын

If you are using Ubuntu 22.04 and missing memory stats in grafana to monitor your docker containers, you'll need to change the version tag on cadvisor to: v0.44.1-test

@pavelmishin3211 Жыл бұрын

Very useful tutorial, thank you!

@christianlempa Жыл бұрын

Glad it was helpful!

@FrederickAlvarez_ 2 жыл бұрын

I love your content, you the teacher I never had

@christianlempa 2 жыл бұрын

Thank you so much :)

@lchig0 5 ай бұрын

Great video akhi

@christianlempa 5 ай бұрын

Appreciated

@qoutwest 2 жыл бұрын

Great video! Going to try and set this up now that I have my Proxmox server up and running

@christianlempa 2 жыл бұрын

Awesome! I'm currently working on getting this running on proxmox as well 😁

@keyshine5769 2 жыл бұрын

@@christianlempa can you please make another full video when you got your proxmox up and running especially monitoring on LXC and etc. Im sure it will benefits alot of us!! 😭😍

@christianlempa 2 жыл бұрын

@@keyshine5769 I guess that I will replace Proxmox at some day and focus more on Docker than LXC, there will be a big change in my Homelab next year 😁

@Rfeliz456 Жыл бұрын

Great Explanation sir, you have earned another follower :)

@christianlempa Жыл бұрын

Welcome aboard!

@kanstantsinkatlinski8502 Жыл бұрын

Dude! It helped a lot, many thanks!

@christianlempa Жыл бұрын

Glad you liked it!

@alemran7480 Жыл бұрын

Thank you so mush ❤ Fantastic tutorial 👏

@christianlempa Жыл бұрын

Thank you so much ;)

@raoufgeorgemdimegh2781 2 жыл бұрын

thank you for this amazing video very helpful 👍🏻

@christianlempa 2 жыл бұрын

Thanks!

@hanpoyang Жыл бұрын

Thanks for sharing, that's helpful.

@christianlempa Жыл бұрын

You’re welcome ☺️

@UltimateKeyboardHero 2 жыл бұрын

I love your two recommended dashboards! was using statefulset daemonset metrics board before, but that's so bad compared to the node exporter full!

@fabricioncosta 6 ай бұрын

excelente material!

@christianlempa 6 ай бұрын

thank you!

@Draugr793 2 жыл бұрын

Wow. Thank you very much. That was great.

@christianlempa 2 жыл бұрын

Glad you enjoyed it!

@packetattack-netsec Жыл бұрын

Great tutorial, thank you! For any beginers out there. if your like me.. im trying to learn YML syntax... Linux and docker/portainer all at the same time.. Make sure your syntax in your prometheus.yml file is correct or else it wont start... I also had to identify the user: root in my stack editor for prometheus so that /etc/prometheus/prometheus.yml could be read.. im sure thats not secure.

@neuralnetdev Жыл бұрын

the volumes that your container accesses, run this command on them: sudo chown -R 65534:65534 prometheus_example/ replace "prometheus_example" with the folder name that you provided to your volume(s) 65534 is the user and group id of the user/group that runs prometheus in the container. for me, grafana also had permission issues in the logs of the container, too. the user and group id is 472 so you'd replace "65534" with "472" in the command above.

@soukainaboualy912 3 ай бұрын

Thank you so much, it's so understandable nd clear

@robertoiglesias5427 6 ай бұрын

Thank you bro, 2 years old but still extremely useful.

@christianlempa 6 ай бұрын

Thank you! Glad it's useful :)

@AiJogja 7 ай бұрын

this is the great video, very helpfull. thanks man.

@christianlempa 7 ай бұрын

Thanks

@edisvila 2 жыл бұрын

Great video as usual. There's just one thing though that I'm curious about and that is missing here. Notifications, for example when a service goes offline or when a specific event happens. I've seen that Grafana has an Alerts feature. I would really appreciate if you could make a video on just something like that for real life scenarios where a notification is useful

@christianlempa 2 жыл бұрын

Such a great idea! I'm planning a new video about that, but that's a bit down my list for next year.

@edisvila 2 жыл бұрын

@@christianlempa Looking forward to it :)

@YouTubers-rj9xv 2 жыл бұрын

Bro excellent teaching

@christianlempa 2 жыл бұрын

Thank you mate! :)

@sangtengkorak1954 2 жыл бұрын

+ Good video with deep thoughts as usual

@paulsalele3844 2 жыл бұрын

Awesome content!!

@christianlempa 2 жыл бұрын

Glad you think so!

@scholziallvideo Жыл бұрын

hi, perfect video. i use prtg monitoring for the servers etc. Primary windows servers and vcenter hyper-v etc.

@user-tl4nq3ib5g Жыл бұрын

thank you for your work

@christianlempa Жыл бұрын

You're welcome :)

@malyyigor34 5 ай бұрын

nice, thank you

@christianlempa 4 ай бұрын

you're welcome :)

@1Chitus Жыл бұрын

Great video ! Just one tiny question: I intend to install Prometheus and Grafana on a RPI4 with 2GB of RAM. Is it enough to monitor about 5 containers and system metrics ?

@djKenpLan09 8 ай бұрын

Me -> where can I find a good tutorial about Grafana and Prometheus? Christian -> here you go! :D Thanks for all you do for the community!

@christianlempa 7 ай бұрын

Haha thanks 😊

@sarmadahsan869 Жыл бұрын

amazing, Help me a lot. Thanks

@christianlempa Жыл бұрын

You're welcome!

@haoxuefeng9336 2 жыл бұрын

thanks for the amazing tutorial! I've followed step by step and it worked on my local machine really cool. I'm curious to know if it can also monitoring remotely for a server? Any suggestions?

@christianlempa 2 жыл бұрын

sure, the metrics exporter works over network protocols, so you can just enter an IP from a remote server. But make sure you're protecting and authenticating the connections!

@nguyendhn 2 жыл бұрын

Thank you so much! It's so easy to understand and keep following.

@none_the_less Жыл бұрын

Well done sir.

@christianlempa Жыл бұрын

Thx :)

@hetzandreas3442 2 жыл бұрын

2021 and you show grafana ? Thats "excellent"

@Tyrun101 8 ай бұрын

Thanks!

@christianlempa 7 ай бұрын

Many thanks for your support 🥰

@christophercarr4207 Жыл бұрын

thank you

@NemesisKo1 2 жыл бұрын

awesome

@daiben2004 6 ай бұрын

Nice Explanation, It will be more beneficial if you set up Alert manager, How we can do?

@mohammad-aminebanaei886 2 жыл бұрын

awesome thank you

@christianlempa 2 жыл бұрын

You're welcome

@hcrp8840 2 жыл бұрын

thank you bro very awesome thank you again thanks a lot

@christianlempa 2 жыл бұрын

Thank you! 😉

@fadjarpradja 2 жыл бұрын

awesomee !, thanks

@christianlempa 2 жыл бұрын

You're welcome!

@huseyinbarin1653 2 жыл бұрын

awesome!!

@christianlempa 2 жыл бұрын

Thanks!

@joshuaplouffe1439 2 жыл бұрын

Could you do a video on how to set up alertmanager on top of this setup? Its a missing piece to this AMAZING tutorial!

@christianlempa 2 жыл бұрын

Great suggestion, however I'm still working on so many projects. But at some point I'll make a video about Grafana Alerts maybe, we'll see ;)

@conradtaylor29 Ай бұрын

Hey Christian, this was a fantastic video and I learned a ton. I was wondering, how did you setup Portainer?

@christianlempa Ай бұрын

Thank you! I think there's a tutorial for portainer on my channel :)

@conradtaylor29 Ай бұрын

I just ended up adding Portainer to the docker-compose.yaml file and this appears to work.

@DeveloperAnand Жыл бұрын

Thanks

@christianlempa Жыл бұрын

Welcome

@surisurendrababu 6 ай бұрын

Hi can you make a video using elasticsearch and grafana ?

@mohamedhaneef6379 Жыл бұрын

Awesome!. I would like to work on a solution that would visualize various metrics from Palo Alto firewalls. I saw an exporter for Fortigate firewalls. Could you please suggest a way forward? How would I customize an integration option for Palo Alto firewalls? Thanks:)

@amsirajuddin 2 жыл бұрын

nice one!

@christianlempa 2 жыл бұрын

Thx!

@RameshSingh-oj7hq Жыл бұрын

Thanks for making the video can you please let me know how Prometheus monitoring is different from Splunk monitoring

@alaahaider 2 жыл бұрын

Great video, very useful. Thank you. Would it be possible to monitor different users login on grafana? Also would it be possible to have a summary of all servers on one page? So you can have a full overview of all your system? Thank you in advance

@christianlempa 2 жыл бұрын

Grafana is just a visualizing engine, so for monitoring you'd need a system collect and aggregate the metrics and later use grafana to visualize it.

@alaahaider 2 жыл бұрын

@@christianlempa thank you so much. You are awesome and your videos are awesome 😎

@cybersecurehacks 2 жыл бұрын

Nice. I am gonna implement this tomorrow in my organization. Also is there a way to manage esxi storage from some 3rd party tool?

@christianlempa 2 жыл бұрын

Awesome! Unfortunately I haven't worked with esxi storage lately.

@mohammadhadi8142 2 жыл бұрын

Perfect

@christianlempa 2 жыл бұрын

Thx!

@samdhillon8325 10 ай бұрын

Nice tutorial as always!! Thank you Christian. I have tried to follow your steps to setup Prometheus but for some reason its is not reading the configuration file "prometheus.yml" from the host instead reading it from the prometheus container which i don't want. Can you please help me figure out what I may be missing here :-)

@christianajeitoh6533 3 ай бұрын

Thanks very much for the content. It's very informative. I have a question. If I have n nodes in my cluster. will node exporter give metrics for all of these node machines OR do I need to run node_exporter on each on these nodes

@christianlempa 2 ай бұрын

you have to run node_exporter on all nodes as far as I know

@mohammedsnoussi8782 Жыл бұрын

Grafana dashboard layout displays all servers as IP addresses which is difficult for us to keep track of and we prefer to have it as a hostname display in the drop down. Any suggestions on how to get that done would be very helpful.

@SravanKumar-bi9mw Жыл бұрын

very useful and Great video - I have done same process and dashboards got visible:-) But i got additional requirements to monitor ,Linux OS version. Which i am struggling and not showing with this ID 1860 ,Moreover, I am using node exporter +Prometheus data source. Can you please suggest on this.

@vladx71 Жыл бұрын

It was great! There is something I'm trying to solve for a long time. For log collection I'm using Graylog, but I always facing issues when I try to implement this as Docker containers. Do you have a working Graylog compose file works in Portainer? I think you should consider a video on Graylog because it is a great and stable tool too and as you explained, logs are also important :)

@christianlempa Жыл бұрын

I haven't looked into Graylog, yet :/ But I'm planning more monitoring videos in the next year, there is definately an update needed!

@KrakPoT13 9 ай бұрын

I'm very new to this -- I see that both containers are separate now. How would I go about deploying them now?

@zag1964 2 жыл бұрын

Just came across your video. Great content. I do have one question. Is it possible to use the third-party exporters to fetch data from a hypervisor server while running in a docker inside a proxmox VM? I am running Proxmox in my home lab and have created a ubuntu server VM and I am running docker in the VM. While your tutorial works great for obtaining data from the host VM, I am more interested in the bare metal data running Proxmox. I am guessing if this is not possible I would need to run the docker in the ProxMox install directly on bare metal?

@christianlempa 2 жыл бұрын

Great question, it is possible with the proxmox pve exporter: github.com/prometheus-pve/prometheus-pve-exporter Might be a bit awkward to setup, but it's working on my end.

@Zero-qs8uq Жыл бұрын

Hey! Thanks for your video 😄 I have a question: There is a way to use Prometheus Cloud + Grafana Cloud without self-hosting it, if yes, then how? Thanks!