Thansk for following along with me, and I love to have folks learning with me!

Thanks so much, and glad it's helpful!

Absolutely, if you're already using a tool for backups, adding this to that workflow should be pretty straight forward I hope.

I've been using timeshift for my servers back ups.. for some reason it wont back up my docker folders... i believe timeshift uses rsync.. So maybe i'll do this ^^ instead.

First, I'm sorry that happened to you at all, but I'm glad you are ok, and what a great testament to the value and importance of backups, but also the easy of use of Docker for quick recovery.

Great tip! Didn't even thing of doing it in a loop.

@@AwesomeOpenSource #!/bin/bash # Specify the root directory where you want to start the loop root_directory="/var/dockers" backup_folder="/var/lib/docker/backup" backupDate=$(date +'%y%m%d%H%M%S') ipAddr=$(ip addr show $(ip route | awk '/default/ { print $5 }') | grep "inet" | head -n 1 | awk '/inet/ {print $2}' | cut -d'/' -f1) fileName="$ipAddr""_""docker_backup""_""$backupDate"".tar.gz" # Loop through each subdirectory in the root directory to stop Docker container find "$root_directory" -mindepth 1 -type d | while read -r directory; do echo "Processing directory: $directory" # Run your script here # For example, if your script is named "myscript.sh": # ./myscript.sh "$directory" # Add your script execution command here cd $directory docker compose stop echo "Finished processing directory: $directory" done cd $backup_folder tar -czvf $fileName /var/lib/docker/volumes/ # Loop through each subdirectory in the root directory start docker container find "$root_directory" -mindepth 1 -type d | while read -r directory; do echo "Processing directory: $directory" # Run your script here # For example, if your script is named "myscript.sh": # ./myscript.sh "$directory" # Add your script execution command here cd $directory docker compose start echo "Finished processing directory: $directory" done

Nice. I have Restic on my list to look into. Thanks for the info!

I learn something new every day, and you sir have absolutely taught me something new. Thank you!

My pleasure.

Thanks.

Thanks. I added that to my shownotes, just not in the video.

Awesome tips! Thanks for that!

I hope it will help.

@@AwesomeOpenSource Running bash script like this gives you less option for remote backups but if you want to save vm resources from continuous usage, your method is effective. Thank You. Big Fan

My pleasure.

Great tip! Thanks for that!

Why don't you stop the docker service? (systemctl stop docker docker.socket).

@@nicolasotero6424 About a year ago when I asked for a solution to stop all containers that what I was given. It works so I use it...

Great video. I am new to all this. Question how would you automatic this at 3 am when it asks for you Suso password when you run your script. Thank you

@@nigelholland24 You can solve it using "public and private keys"

I'll do a follow up and talk about that part. It's a one liner in the script. I run it at the beginning of the script to make room before I run the backup and move it to long term storage, but you can really run it anywhere in the script.

Very nice. I've not tried Restic but it's on my list.

Totally valid things to do. You can definitely cut down on size by excluding logs, tmp, images, and so on that can be easily rebuilt / re-pulled from some other location if needed.

@Awesome Open Source yep. Also, my backups are in 2 locations. One in a crypt and the other on an older server that can take one some of the tasks in case my main server goes down

I also like to suggest using Restic or Borg instead of tar. They are incredibly more space efficient and can even mount the backup via FUSE so you can do file restoring.

I actually do backups of all my VMs every night, and move those to my NAS. I just wanted to show how to backup Docker specifically for those that may not have that ability or functionality today. But 100%, those are all fantastic backup options.

You bet.

Sure. Let me try to setup a test system to restore to. The thing about the resotre is if you have NGinX setup for apps like I do, then you have to change those IPs for the different machine, whereas in the case of a failure, in theory you should be able to assign the same IP to the new machine. Just FYI, I have done this before, I literally hosed a VM, and used the backup that I had from this same method, and was back up and running in less than an hour.

Nice. Maybe you'll open source it, and I can cover it on my channel one of these days!

Absolutely. There are tons of options, just pick the one that works best for you, and grow from there.

Now you're moving toward clustered containers (Swarm, Kubernetes, etc) so you backup from one node at a time, and update in a staged manner that rolls through so there is no perceivable down time.

@@AwesomeOpenSource A video on that would be really great. I'm about to spin up my third Portainer node. I would love to learn how to cluster them.

@@AwesomeOpenSource I have two more machines coming in the mail. Looking forward to building a cluster next week. I'm very confused about your backup script however. If you have a moment, can you help me understand? Why are you stopping each container individually like that? Why can't we just shutdown compose and docker entirely with two simple commands. Or even one simple command to simply stop docker? The way this is scripted, I'll have to change the backup script every time I add or remove a container, right? Am I misunderstanding?

That's a great way to handle it. I do something similar on my Proxmox setup. I actually backup my CTs and VMs each night to my NAS.

Agreed, if portainer could do the volume backup and config / compose backup, it would be the perfect tool for this.

make next step and use swarm with stacks

Let me see what I can do.

@@AwesomeOpenSource ur awesome brian , il be waiting amazing videos and content

@@AwesomeOpenSource please reply.... i deployed my stuff using stacks/docker compose files and when i try to run "docker compose stop" eg for jellyfin it doesnt stop Please guide what should u do to make the script work

If portainer is managing those containers it may be a little different, but maybe try just using "docker stop jellyfin" and see if that works instead of using the docker compose command.

I separate mine just for the organization I showed, but also so I can run docker-compose pull without having to specify different applications, which could be disastrous with all of them in one file. I'm sure there are mitigations either way, so just make sure you are good with how it all works with everything in one file.

Still no encryption though sadly

Very cool.

For encryption of the backups using my method, you can store the final zip file in an encrypted partition or vault for sure.

For production, a tool like Kubernetes would probably be ideal, and then pulling down a node on the cluster for a backup is likely less interruptive. I know there are some specific tools for doing these types of things for Kubernetes as well.

No, no need to backup the inner part of the container. It will pull down and startup on a new system as needed, so you only need to backup data you want to persist, and your compose file just as a reference if you need to bring it up fresh on a new system.

@@AwesomeOpenSource thank you very much!

So, if you don't specify that the mapped volume should be in the current folder (the ./ portion of the left side of the mapping), then docker generally puts the files in a standard place, but that place will potentially differ from distro to distro. You just need to find that place, and backup those volumes from there.

I don't back up all of my containers. Only most of them, so for me it's easier to edit and update the script as I have it, but your method would probably work fine.

I have one on pfSense and DD-WRT VLAN setup now. Is that not what you are looking for?

#!/bin/bash FOLDER=/home/$USER now="$(date)" echo "Stopping containers started at $now, YAY!" docker stop $(docker ps -a -q) sudo tar -cvzf $FOLDER/backups/docker.tar $FOLDER/docker/ echo "Restarting containers started at $now, YAY!" docker restart $(docker ps -a -q) That's what I have been using for a while now

@@HelloHelloXD I added an exclude file to exclude a few folder with content I have backed up elsewhere. (All the book and comic bundles from humble bundle. They really do take up a lot of space and make the backup way too slow🤣) static content I can easily copy back from the source etc

@@HelloHelloXD Also, for those using (non-rootful) podman, the script should largely be the same except replace sudo tar with "podman unshare tar" so it will perform the backup using the podman userspace. Also obviously replace all the "docker" instances with "podman" too

As long as it works, then well done my friend!

Very nice!

You can leave off the 'v' in the flags for the tar command, so it would be 'tar -czf' instead. The 'v' means verbose.

@@AwesomeOpenSource you are a champion. I am now using your script to backup volumes created in portainer.

I was just trying different things as I was making the video, and the last one was scp. Rsync works well too, but in honesty, since we are talking about an archive in my case, the rsync won't do an incremental backup, but would just move the new file over (same as scp).

It can, for sure. But generally it's a fairly fast operation. The alternative is to add a db dump to the script. A bit more involved, and depending on the database, maybe a bit more tricky to restore to a running container (mongodb, I'm looking at you), but it is an option.

I believe you'll do it through the AIO interface, but I haven't had to restore yet.

@@AwesomeOpenSource how about ssl ? will it reconfigure automatically??

This is one of my only videos where I did any presentation. I just felt it was a simple way to get the points across before getting into the work, and that wasn't PowerPoint, it was LibreOffice Impress (open source).

@@AwesomeOpenSource I wasn’t criticising:) 0.0 criticism here, just the joke. Loved it. Continue however you want to and I’ll watch. Thanks for inspiring and point towards meshcentral, that’s saved my buttocks multiple times.

It's not better than any backup tool. The important information to have is that when backing something up, you should try to make sure there is no writing of information during the backup process. If you can schedule that to coincide with the duplicati backup, then duplicati is a perfectly fine tool for the backup portion. I'm just trying to show folks how to 1. stop their containers, 2. create an archive to save space, and 3. restart their containers. The last step of backing up the archive can be done with a whole host of tools.

You are right. In this case, I'm not saying to make a db dump (proper) backup, but instead, talking about fully backing up the mapped volume where the db data is stored for the container, and in that case, we don't want writes to be happening when we make that copy if at all possible.

I'm sure there are lots better options out there. It suits my needs, and is a place to start for anyone who wants to backup their stuff. There rae full programs out there that do this far better, no doubt.

I definitely need to check it out further, a few have suggested it.

I am interested. How does your env. look like

I saw that option, but wasn't sure it did everything I wanted, which is configs, volumes, and compose files.

I like where you're going, but I would definitely have to research this more to fully understand it all.