Get Docker organized for easier backups & replication. Trust me, an hour can save you days!

Рет қаралды 12,146

Күн бұрын

I recently had a server go crazy and just put up the full iptables firewall blocking all inbound and outbound network connections. After a week of trying to fix it, I decided to rebuild a new vM and move my dockeer stuff over. This video is about how I did that, and how my docker organization helped save my tons of time and grief. I hope it helps you, and I hope you get something out of it to save you from making a similar mistake.
=== Links ===
Support my Channel and ongoing efforts through Patreon:
www.patreon.com/bePatron?u=23...
=== Timestamps ===
00:00 Beginning
00:09 Introduction
02:30 Thank you to my Patrons over at Patreon, and to all of my KZfaq Subscribers!
03:05 My Docker and VM Setup Diagram
07:12 Organizing your Docker Setup
08:45 Making docker-run.txt files
11:40 Keeping Docker Folders with their Data and run or docker-compose files
15:30 The use of sub-folders for categorization
=== Contact ===
Twitter: @mickintx
Telegram: @MickInTx
Mastodon: mastodon.partecipa.digital/ @MickInTX
Try out SSDNodes VPS Services! Amazing Specs for incredibly low costs. I'm running a 32 GB RAM / $ CPU Server for only $9 a month! Seriously. FOr long term server usage, this is the way to go!
www.ssdnodes.com/manage/aff.p...
Get a $50.00 credit for Digital Ocean by signing up with this link:
m.do.co/c/a6a61ae55242
Use Hover as your Domain Name Registrar to get some great control over you domains / sub-domains:
hover.com/SHPaiirr
Support my Channel and ongoing efforts through Patreon:
www.patreon.com/bePatron?u=23...
What does the money go to?
To Pay for Digital Ocean droplets, donations to open source projects I feature, any hardware I may need to purchase for future episodes (which I will then give to a subscriber in a drawing or contest).
=== Attributions ===
Intro and Outro music provided by www.bensound.com

Пікірлер: 50

@happysprollie 2 жыл бұрын

This is why I like docker-compose, because it's easy to recreate containers and is practically self-documenting

@AwesomeOpenSource 2 жыл бұрын

Absolutely! Compose is a tremendous tool.

@default_youtube_profile 2 жыл бұрын

But docker-compose is not docker swarm

@froid_san 2 жыл бұрын

As someone still learning the in's and out's of docker, this is a very great tip. Luckily since I'm still a newbie, I've been doing all things docker using docker compose and always saving it as I'm afraid to mess up. Now I must keep in mind the data files of the containers to backup. Great vid!

@AwesomeOpenSource 2 жыл бұрын

Thank you, and it sounds like you’re on the right track.

@MmMm-oh5ct 2 жыл бұрын

Its Awesome, I am on my way to Setup kind of something with ansible at the moment. Great Video

@AwesomeOpenSource 2 жыл бұрын

Thank you.

@microcisco Жыл бұрын

Great channel and love your videos! I know most people don't have the infrastructure to be able to run a hyper-converged setup, but if you have 3 physical machines and SSD disks to run a Ceph storage (storage as a service - StaaS) on 10Gb interfaces. I run a Proxmox 3 physical server setup with the 3 in a cluster and Ceph Storage to form the HCI. I run the Proxmox backups nightly and it has saved my ass a few times and easily restore the backups to any of the 3 in the cluster with all attached to the Ceph storage. If you can setup the HCI environment it will give you peace of mind and the functionality to easily restore failed VM's and containers. (As a side note, I used to run LXC containers but I find it much easier to run VM's with docker on those VM's.)

@AwesomeOpenSource Жыл бұрын

Great information and advice. Thanks for sharing.

@JPEaglesandKatz Жыл бұрын

Also the way I've set it up is bind mount every single data folder from any container I use to a cental data folder... I backup this folder with borg backup and also create periodic toal VM backups, including syncing them to an external drive as oposed to only my zfs pool.

@AwesomeOpenSource Жыл бұрын

Nice. Great tip!

@kevinhughes9801 2 жыл бұрын

Great stuff thanks

@AwesomeOpenSource 2 жыл бұрын

My pleasure

@CaptZenPetabyte 2 жыл бұрын

Great idea as Im about to start a new project & setting it up well like this from scratch will be so much better. Could prob even write a bash.sh to copy & bring everything back up again to automate

@AwesomeOpenSource 2 жыл бұрын

Better to start off organized, but worth going back and re-organizing as well!

@rockettpc2 2 жыл бұрын

Great info, funny, I kinda intuitively started like this, but I find myself doing the same in my individual devices, so it was kinda the norm for me. But this is very important to do, stay organized and have backups etc. I am playing with docker swarm on a 6 Node Odroid C2 cluster and I'm using NFS for the data, I really need to build a good NAS ideally more than one for redundancy of course, but it's all allot of fun self hosting and pushing the edge of home computing.

@AwesomeOpenSource 2 жыл бұрын

Keep it going, and glad you are already on track with organization!

@kronos3428 2 жыл бұрын

Thanks for the tutorial. This has always been a big concern of mine, I like to do things just once. I recently started using TrueNas Scale which is debian based and has docker support and a very interesting thing I found out is that you can change docker root folder, so I have this disk that when attaching it to any other instance, the whole date the containers are there and also the containers, they even start automatically (following the rules previously assigned).

@AwesomeOpenSource 2 жыл бұрын

Nice setup.

@MrNoBSgiven 2 жыл бұрын

Great topic. 1. Backups is a must. Snapshots of VM would have helped you recover in no time at all 2. . Portainer has a backup function,mwhich backs up the entire setup and allows recovery to a new machine in no time at all. Finally, have you tought of automation of the backup and recovery/seting up using terraform (proxmox VMs) and ansible for the "fresh rebuild" of your VM environments? THESE WITH distributed file system storage could help. I am in the process of setting something like that. I am thinking about using docker swarm to take away the headache of manual splitting of docker apps.

@AwesomeOpenSource 2 жыл бұрын

Agree 100% on the backups. And i do, now, have a backup of my VM, and it's automated as well. The docker part is an interesting one. I did find the Protainer backup (after the fact), but will start using it too...yay! As for Swarm, I'm not there yet, but was also thinking this would be a great reason to learn and start using kubernetes at some point. As it would have provided the no-down-time reliance I might want some day, while I'm re-building one server if needed.

@gravyflex 2 жыл бұрын

I'm the middle of setting up a docker swarm built with terraform and Ansible. Working on NFS persistent storage with a docker plugins. Right now I am figuring out traefik reverse proxy.

@artedwards717 Жыл бұрын

Hey brain awesome I wish you could make a drive with that all on it I am learning wow on your videos

@AwesomeOpenSource Жыл бұрын

What do you mean by a drive?

@artedwards717 Жыл бұрын

@@AwesomeOpenSource oh brain I mean like a CD to load on my main computer

@JPEaglesandKatz Жыл бұрын

checkout borg backup.. its great.. easy to put into a cron job, backup up root owned folders, incrementalized backups, compression, protection, etc... set and forget once setup correctly

@AwesomeOpenSource Жыл бұрын

I've looked at Borg a few times, just haven't tried it yet. It's on my list though.

@JPEaglesandKatz Жыл бұрын

@@AwesomeOpenSource Cool... Yeh it is pretty good... Keep on rocking!! I love your channel, came across some really cool pieces of software that way... I was a linux noob 8 months ago... now it is starting to feel more like home than windows!

@SaroxDuPoney 2 жыл бұрын

Hello thanks for your tips. What is your container for your dashboard @16:07. I like it a lot.

@AwesomeOpenSource 2 жыл бұрын

Currently I'm using Homer, but just recorded a new video on Dashy, and I'm thinking about switching to it. I need to test it for another week or so before I decide. Video coming soon!

@SaroxDuPoney 2 жыл бұрын

@@AwesomeOpenSource ok thanks. I will be looking for your next video. Thank you very much for all your info.

@markstanchin1692 2 жыл бұрын

Hi, I just ran your latest script, how cool! I’m rebuilding my system in a new VM and docker etc. trying to get organized the way you show in this video. The script helped. A Couple of things I noticed it does not create a docker folder for the containers it installs. Also it says it installed Portainer agent but it’s not listed under the containers with port 9001. Also the stacks that are created say they have Limited management under Portainer. Any thoughts? Will you be adding other applications to the script? Also when I click on a port from a container it takes me 0.0.0 local host I remember somewhere how to set this to the IP but I can’t find it. Thanks ! now I’m back to work .

@AwesomeOpenSource 2 жыл бұрын

Yeah, I need to check for or create the Docker folder, not am not doing it today. Not sure why the Portainer container isn’t shown if it installed properly. The IP is set from your main Portainer Home, then I believe under Environments. I’m considering more options for containers, but at some point I’m just duplicating what Portainer can do in the UI. As for the limited management, I thinks that’s what it always says for Docker-compose created outside of the Portainer UI.

@50_Pence Жыл бұрын

Did that mariadb backup work OK? Or did it corrupt? Did you have to stop the container first?

@AwesomeOpenSource Жыл бұрын

If you have a system that writes to MariaDB a lot, then it is better to stop the container, then do the backup, and start it back up. I've been lucky so far, and my containers are generally low usage, so when the automatic backups run, no one in my home is up and using anything.

@majidkhoram2451 2 жыл бұрын

Did you know you are brilliant? Or should I mention it?😉👍🏻

@AwesomeOpenSource 2 жыл бұрын

I appreciate the compliment, thank you!

@markstanchin1692 2 жыл бұрын

Ok i’m in trouble. So I installed a bunch of docker containers using different methods portainer, docker compose etc. When I ls under my home directory I only see two dirs , Heimdall and guacamole soo I thought when you installed a docker container it installed and created the Paths like in your example automatically. There installed and working perfectly but I have no idea where the dockers Containers and data is installed by default. So without redoing like 15 containers is there a way I can find out we organize and regroup the way that you’re showing in this tutorial? Thanks

@AwesomeOpenSource 2 жыл бұрын

When I look at my file system, I find Portainer stuff under /portainer/Files/appdata/ , so you may start there. There is also stuff under /data/compose. You can use the Portainer UI to backup your docker stuff as well. But if you want to organize it, you'll need to move stuff from the file paths above, and then start cleaning it up. Do one at a time, take your time, and make sure each app comes up as you expect.

@markstanchin1692 2 жыл бұрын

@@AwesomeOpenSource Thanks! Are you going to create a video on how you actually do this and how you back up your folders zip up script etc? Also I was thinking of moving data from the containers over to a Nas device not sure if it makes sense or how it would work have you ever done anything like this? Also off-topic I host a couple of websites not WordPress on site ground. Well the renewal is coming up and it’s four times my original price. So I’m thinking of moving to self hosting my sites but I like the way site ground works with the control panel. Is there anything that can be installed on a Ubuntu vm server that can act as a control portal like cPanel? I’m thinking of creating a separate VM just to host my websites does that make sense?

@lezz27 2 жыл бұрын

Nice video. I just had a crash and I know the pain in getting everything up and running. I have a mic of Windows and Linux machine. My Docker stuff runs on Ubuntu 20.04 where as I have shares on my Windows machine. I was wondering whats the best way to backup the docker files onto the Windows share. You mentioned something abt the script you use to zip and upload to a share. May you please share the script? I tried using Duplicati and Duplicacy on my Windows machine and then mounted the Docker volume on Windows machine using "Mountain Duck" application. The issue that I am facing is that there are some parts of the Docker folder that I am not able to backup using these backup apps due to permission issue. I am not sure why I have that. I tried to grant my local user on Ubuntu a root access, that didn't work either. Any help/suggestion in this matter will be appreciated. Note: All my shared folders are on my Windows machine.

@AwesomeOpenSource 2 жыл бұрын

Couple of things to look into. 1. make sure your user on ubuntu is part of the docker group. sudo usermod -aG docker {$USER} 2. Setup a Samba share, and make sure you mount it on your Ubuntu server. 3. Now you can zip your docker folder ( you may have to use sudo ), then copy that folder to your mounted Samba share. When doing this, you'll want to make sure and delete versions over time so you don't fill up your drive on the Ubuntu machine. The script i used, 1. created a gzip file of my docker folder, then 2. rsync'd the file over to my backup server, and then 3. deleted the gzip file. So each day the files were zipped, backed up, and then the zip was removed. It was a very simple script, but effective.

@lezz27 2 жыл бұрын

@@AwesomeOpenSource This is the quickest response I have every seen in YT. Thank you! I am bad with writing scripts. Would you mind sharing your script? I did setup my shares on windows with required permissions and currently trying to figure out how to mount the Windows SMB shares on Ubuntu server.

@dimitristsoutsouras2712 2 жыл бұрын

10.45 Pleaseeeeeeeeeeeeeeeeee pass the link of that procedure. If it is what I think, I am looking for this for months. My use case scenario is..... in a win2019 vm which acts as n A.D and have been installed on a raid10 zfs storage (storage derived from the same server the VM is installed)I have attached an extra portion of another raid10 zfs (derived from the same server as well).....what I left thinking afterwars is... if I migrate the VM what happens with that extra disk I have attached to that VM since it is not a shared folder or iscsi... it was just a block device that windows format and used. I m stuck there for ages!!!

@AwesomeOpenSource 2 жыл бұрын

Instead of a script, I've found a great container called Duplicati and just recorded a video on it, as well as how to use it to backup your docker folders (or anything else) on a nightly schedule. It's super great. Coming soon!

@dimitristsoutsouras2712 2 жыл бұрын

@@AwesomeOpenSource I already know about Duplicati and UrBackup and other programs as well. I have to wait for your video and see if it's use case scenario can be applied to me too. PS By the way for me it would be useful a program that can take a block level backup and not file level.

@AwesomeOpenSource 2 жыл бұрын

@@dimitristsoutsouras2712 Timeshift on Linux can do Rsync backups / snapshots, and does diffs between the files, instead of complete files. Not block level exactly, but may help. I believe clonezilla can do block level, but you have to boot into Clonezilla to do it, or run it as a VM.

@dimitristsoutsouras2712 2 жыл бұрын

@@AwesomeOpenSource Thank you for your answers. ..... and that is exactly what I do. I use rescuezilla (even better than Clonezilla) and backup each disk (it is a mirror), do the upgrades or whatever I need to and then if anything goes very wrong, I revert back. Problem is (as you mentioned) I have to shutdown the server boot from usb stick make 2 backups (25min each on a 120gb disk - for the OS) and takes time for a production system.

@AwesomeOpenSource 2 жыл бұрын

@@dimitristsoutsouras2712 is this for Windows or Linux?