Backup Your Docker Server NOW! - Setup and Configure Duplicati

Рет қаралды 49,158

Күн бұрын

In this video we're going to take a look at backing up our Docker server. In order to run our backups we're going to use a Docker container called Duplicati.
We're actually going to setup and run 2 backups in this video. The first will be a local backup on an extra hard drive. The second backup will be a remote backup using a 3rd party service like Google Drive.
We will cover how to RESTORE your backups in a follow-up video. It would have made this video too long and would have tried to cover too many topics for a single video.
Here are some things to consider:
- You MUST use a separate drive for your backups. You don't want your backups to be on your main drive. If that drive fails you will lose your in-use data AND your backup data. USE A SEPARATE DRIVE.
- Your separate drive should be AT LEAST as large as your Docker drive. You can't backup a 2TB drive to a 500GB drive. So make sure your backup drive is larger.
- Normally we don't run containers as root with PUID and PGID, but we're going to run this container as root to help ensure no permissions issues with any part of the process.
- You can mount as many volumes as needed to make sure all of your different folders are backed up.
This is the stack I used in the video: hub.docker.com/r/linuxserver/...
More OpenMediaVault 5 tutorials here:
• Openmediavault 5 (OMV5...
/=========================================/
Remember to leave a like on this video and subscribe if you want to see more!
/=========================================/
#DockerBackup #DockerRemoteBackup
Like what I do? Want to be generous and help support my channel? Here are some ways to support:
Patreon: dbte.ch/patreon
Ko-fi: dbte.ch/kofi
/=========================================/
Here's my KZfaq Merch Store:
dbte.ch/ytstore
/=========================================/
Here's my Amazon Influencer Shop Link:
Amazon: dbte.ch/amazonshop
/=========================================/
Follow Me:
Twitter: dbte.ch/tw
Facebook: dbte.ch/fb
Subscribe: dbte.ch/ytsub

Пікірлер: 108

@GlenBland 3 жыл бұрын

I love this. You have been on a roll lately rounding out the larger picture of self-hosting. Great work.

@DBTechYT 3 жыл бұрын

Thanks a bunch!

@jig1056 2 жыл бұрын

This is great, it worked like a charm for me. Using it to back up the containers running on my PI.. I’m on my running a handful of containers on the device so it’s easy for me to shutdown the containers before creating a backup. All of my other containers are running on a VM so it’s easy to back those up using snapshots. Thanks for making another incredibly useful and helpful video.

@ranieri9409 3 жыл бұрын

Thank you!! I tried so hard to setup duplicati today, but it gaved me always a permission error. Setting "0" and "0" for PUID and PGID was and absolute game changer. I had no idea that root have this values. Thank you, from Italy!

@leoprisionero Жыл бұрын

great video, watching you from Bogota, Colombia. Thanks for all the effort

@jccdsantos 3 жыл бұрын

Fantastic tip for backup. I really like your videos.

@DBTechYT 3 жыл бұрын

Thanks for watching!

@andrut04 Жыл бұрын

Thank you! This will make my life so much easier.

@DBTechYT Жыл бұрын

You are so welcome!

@master6hifi893 3 жыл бұрын

Cool & Fantastic Video.Thanks for sharing.

@jorgegomez374 3 жыл бұрын

great video, as always. Take care

@yachalupson 3 жыл бұрын

Great video - Thanks! I was hoping you'd cover automated backup of docker volume data though. Hard topic to find clear guidance on

@corticalstack 6 ай бұрын

Still works like a charm, thanks!

@DBTechYT 6 ай бұрын

Great to hear!

@markshaz8691 3 жыл бұрын

Awesome tutorial thank you.

@sunil951 Жыл бұрын

Thanks for sharing. Love it.

@hamhumtube Жыл бұрын

I like your content. Thank you for great work.

@DBTechYT Жыл бұрын

I appreciate that!

@kirensuarez434 3 жыл бұрын

thanks to share this content with us.

@deviantstudio Жыл бұрын

i like this approach much more than suffering with cli tools (like rsync) and afraiding to make some typo in file paths or so.

@TimothyApe 10 ай бұрын

There is also a "native" OMV plug-in I used called "USB backup", that is pretty efficient because it compares the changes between the data on the OMV and on the backup storage. I wonder if Duplicati is simply a much more advance backup solution, or is there still any use to use this plug-in (maybe because of the beforementioned efficiency perk)? As i understand, duplicati just copies the full backup everytime?!

@Dan-gy3jt 3 жыл бұрын

Hi. This question would apply to all of your docker how-to's: in OMV with Portainer have you been able to map a volume with a bind mount to a share on a windows server (via OMV remote mount to a CIFS) with rw permissions? There are lots of videos talking about doing that from docker on linux to a linux server, but can't find anything on doing it to a windows server. Getting read permissions seems to be easy, getting rw not so much. Maybe a good video topic? Thanks!

@tunlinaung010 2 жыл бұрын

Awesome!! Thank bro

@squalazzo 3 жыл бұрын

thanks, very useful!

@DBTechYT 3 жыл бұрын

Glad to hear that!

@dnullify100 3 жыл бұрын

If I am understanding correctly, you are backing up only the volume used for the docker container data. Do you use this application to backup your main nas shares?

@fabioamado6725 4 ай бұрын

What about backing up containers with databases? Shouldn’t we stop the container first? Can we automate this? Thank you.

@localadm 3 жыл бұрын

Hi, TCP port 8200 is minidlna. Just in case.

@peterfortyseven8549 3 жыл бұрын

Thanks, this seems better then je built in backup from omv itself. But this is only the container data, how to backup the container settings themselves? The stacks and configuration? Keep going with the good work

@DBTechYT 3 жыл бұрын

You'll have to keep your stacks/configurations stored somewhere else. I actually store my stacks on Github Gists for free.

@mariojau Жыл бұрын

I have a question, in your video you have a folder called /srv/dev-disk-by-label-Files/Configs. You set duplicati config to a subfolder called Duplicati and the source as Configs. What is the content of that Configs folder? Is this the stacks maintained by your portainer? I would like to mimic your setup.

@michaelgaluski9594 2 жыл бұрын

Nice video. Do you happen to know how to wake up a NAS from inside the duplicati container (in a script)? There is no such thing in the container as wakeonlan, ping or etherwake.

@DBTechYT 2 жыл бұрын

I'm not sure how to do that. I'm running this to backup my containers on a Synology NAS, but I just have the NAS running all the time.

@ckourelas 3 жыл бұрын

Happy Friday!! Thank you for your video! I have a request if you can make a video about it. How to set up internet radio AzuraCast on Docker on raspberry pi. Thank you in advance.

@DBTechYT 3 жыл бұрын

I think that would be great, but the problem is that, according to their site (www.azuracast.com/install/docker.html), the system requires: - A 64-bit x86 (x86_64) CPU - 2GB or greater of RAM - 20GB or greater of hard drive space - A computer/server capable of running Docker So I'm not sure that it can be installed on a Pi

@ckourelas 3 жыл бұрын

@@DBTechYT Your right on that but I've read this www.azuracast.com/install/raspberry-pi.html#installing-ubuntu although it states that they will not provide any more support to Raspberry pi. Worth to try :) You are the man you might be able to make it work :) but even if you can't make work, still you are the man you are doing amazing Job! Thank you!

@asimabusallam3147 3 жыл бұрын

thanks

@chinthakamayadunne1 3 жыл бұрын

Great! Did you check the integrity of the backed up files if the source was being used, for example if the files were backed up while the container was up and running?

@DBTechYT 3 жыл бұрын

I'm planning on doing a video about restoring the files very soon

@ultrazSupporter 3 жыл бұрын

I'm also interested in this question. How holistic such a backup system files will be.

@DBTechYT 3 жыл бұрын

I just released a video today showing that the backup worked

@ultrazSupporter 3 жыл бұрын

@@DBTechYT I have an Unraid server. And there's a plugin that copies the Appdata folder. There is a scenario that it first stops the containers and then makes backup time and then runs them again. Here, there are no stopping containers, so I wonder how reliable such a backup can be.

@DBTechYT 3 жыл бұрын

That sounds like a great plugin!! I wish there was something more like that for OMV. The process sounds like it should be just fine.

@cattivello 3 жыл бұрын

Finally Friday.. yes! Nice tool. Question: do I need to have a separate drive for data to be able to back up my server? I use on as well, but I only have one drive. How would it back up my data when all server is in one drive? Would it work anyway?

@DBTechYT 3 жыл бұрын

2:19. Come on, man. It's literally right there. You can do ONLY remote backups if you don't have any other options, but you really should have an extra drive for local backups as well.

@cattivello 3 жыл бұрын

@@DBTechYT sorry, I did not explain well. I meant to say the source I have to back up is the all omv. So.. I have the extra hd for backup destination. But my data to be backed up is the all omv, not a separate data drive to be backed up. I figure I need an extra drive where to out the backup .

@cattivello 3 жыл бұрын

Nevermind. I will test it. What i did not understand is how to configure what to back up out of the omv/server .

@jamesT008 2 жыл бұрын

Great

@andrewpndavies 3 жыл бұрын

Great video. Can I check though - are you using Duplicati to backup Docker volumes that are stored in /var/lib/docker/volumes? I don't use OMV so can't quite work it out.

@DBTechYT 3 жыл бұрын

I don't backup my docker volumes as I've got my configs and data being stored in other folders.

@andrewpndavies 3 жыл бұрын

@@DBTechYT Thanks - out of interest - with Node Red for example, running in a Docker - how would you backup the /data folder that stores all your flows and hard work? Congrats on the Portainer role btw!

@DBTechYT 3 жыл бұрын

You would have to have mounted a data volume as shown on this page: hub.docker.com/r/nodered/node-red You could then backup that data volume using Duplicati

@DBTechYT 3 жыл бұрын

and Thanks!! :)

@andrewpndavies 3 жыл бұрын

@@DBTechYT No worries!

@Glatze603 3 жыл бұрын

Nice video about duplicati, but isn‘t it necessarily to stop docker containers bevor backup its configs in order to avoid corruption of files? Other wise you maybe backup files that are written by its docker container while you back them up. I use a script that stops the containers, then backup the docker files and start the containers again.

@DBTechYT 3 жыл бұрын

I've never run into any issues doing it this way.

@Glatze603 3 жыл бұрын

@@DBTechYT But imagine you use a database in config folder (nextcloud)?!? Never ran into a problem does not mean that this could be one :-) Maybe I am wrong, but what if not?

@DBTechYT 3 жыл бұрын

You may absolutely be right, but I always made sure to schedule my backups to be in the middle of the night when I knew there was no chance of me doing anything on the server.

@MarinJuracic 3 жыл бұрын

Great video! Though I have one question. I have 2 drives attached to OMV. I added extra line with a path for the config folder under the line source for the second drive. Example: /srv/dev-disk-by-label-datadisk2/Config2. What bothers me is that I see that folder in Duplicati but I can't see separate folders inside when I press small triangle icon next to it. I run backup and it does a backup of that folder but when I press restore files I only see complete config2 folder without separate config folders for application data inside, so if I wish to restore only one folder inside I have to restore everything inside which is not very handy cause if only plex config file is corrupted I need to restore every other folder inside (qbittorrent, duckdns....). Can you please tell me how to add that second drive config folder properly? Thanks!

@JohanSmekens 3 жыл бұрын

I do follow

@JirkaLottmann 3 жыл бұрын

When i set new backup to another server in my home via sFTP (SSH) and add correct IP, username and password, then show me error message with "Failed to connect:"

@winecountrygames1859 2 жыл бұрын

Does it remake the files daily? or does it just update missing/changed files every day? Is there a way to set it up to delete source files after a successful upload?

@DBTechYT 2 жыл бұрын

You can configure it a ton of different ways. There's really no way for me to tell you all of them. I really encourage you to install it and check out the different things you can do with it

@0ctatr0n 3 жыл бұрын

Just to check I have this right, this is backing up docker servers _while_ they're running? So like Nextcloud mysql data is being copied here as well as the frontend and docker-compose files?

@DBTechYT 3 жыл бұрын

This is why I run the backups at night when I won't be using the containers. It will ONLY backup your application data, not the docker-compose files. I show in the follow up video how to restore your containers.

@persching78 Жыл бұрын

@@DBTechYT Hi, first of all: you've a lot of great videos! Thx a lot. :) I've an additional question. I run nextcloud on a synology DS220+ with docker and using a mariadb container for the database. Would it be possible to set nextcloud to maintenance mode out of the duplicati container and do also a SQL dump of the database? In my opinion, this would be the best way to backup all my nextcloud data. In the next step I would upload the duplicati backup to a cloud storage via rsync. I planned to backup nextcloud once a week. At the moment I've round about 80GB of data and so the upload is not so easy for me, but if the first upload is done it wouldn't be a problem anymore because the data did not grow up fast.

@FunctionGermany Жыл бұрын

i used duplicati to create backups for many months now. however, i tested restoring the backup way too late (you should do this right away!). it turns out that restoring even small parts (20GB) of my ~160GB duplicati backup takes literally 20 HOURS on a different computer!! this is because duplicati has to do very tedious work in restoring the necessary database. that's why i advise against using duplicati unless you test the "disaster recovery" and you're fine with the restore speeds. it's definitely not for me, if there was an issue with one of my server drives i'd want to be able to restore the important services in an hour at most, which is easily possible if it was just a simple rsync copy instead of duplicati backup.

@RogerioPereiradaSilva77 8 ай бұрын

While I love rsync with all my heart, it should be noted that rsync is a file syncing tool NOT a proper backup per se. If a file is lost, corrupted, etc. on the source, it will be copied "as is" on the destination, rendering it useless from a backup point of view. A decent backup solution should at the very least provide file versioning. If having immediate access to the files matter that much to you, you should consider something like rsnapshot or rdiff-backup that provide all the advantages of rsync while adding useful backup functionality.

@FunctionGermany 8 ай бұрын

@@RogerioPereiradaSilva77 you're right, and i'm not using rsync for backups. i'm using borgbackup/borgmatic.

@novaleary4488 2 жыл бұрын

Can I setup Duplicati to use only cloud backups? I dont have an extra 4TB drive to backup my stuff and just want to keep the backups of my nas on a remote server (with Backblaze)

@DBTechYT 2 жыл бұрын

You can. I'm actually going to be doing a video on how to use Duplicati to backup to BackBlaze in the near future

@novaleary4488 2 жыл бұрын

@@DBTechYT Thank you for the response! I ended up switching to RClone because Duplicati seems to want to create a database on my raspberry pi and it quite literally ends up eating all of my 64GB of space on the sd card.

@ronald0122 3 жыл бұрын

Can i do this while the containers are running?

@DBTechYT 3 жыл бұрын

Yes

@redlingg 3 жыл бұрын

Hello. I have installed this in docker but when I want to choose the directory to back up(/var/lib/docker) it does not see it..(It is there...) Any suggestion?

@hathalud 3 жыл бұрын

Add a mounted volume for that directory path so that Docker will allow Duplicati to see that directory outside of it's container.

@Varian-Killbot 2 жыл бұрын

I should have known this would be too easy. When I got to test the connection after generating an AuthID, I get a "Failed to connect: Internal Service Error". Flipping the slash as some suggest online just gives a "Bad Request" error instead. Edit: Solved. Had to go to their appspot page, generated a full access token, further had to manually create the drive folder from the site itself, THEN it works.

@DBTechYT 2 жыл бұрын

Great job!

@itsathejoey Жыл бұрын

Could you backup the entire root file system with this method?

@DBTechYT Жыл бұрын

Maybe. Never tried.

@TheDalejake 10 ай бұрын

I keep getting an error when trying to access the web gui. it's using http instead of https . very new to all this and don't know where to look to correct it

@DBTechYT 10 ай бұрын

I would try using a different browser then. There's nothing in the docker-compose that uses https. In fact, the notes on the hub.docker.com page explicitly state that port 8200 is the http gui port

@ankurpatel8927 3 жыл бұрын

How are you resolving the hostname? Are you using Windows Active Directory? I'm trying to find a solution without using Windows.

@DBTechYT 3 жыл бұрын

I have it set in Openmediavault, but you can set a hostname in any version of linux.

@ankurpatel8927 3 жыл бұрын

@@DBTechYT what is resolving your IPs? My router isn't resolving my IPs so I'm looking for an external solution.

@jepolch 3 жыл бұрын

For some reason I can't figure out, my server is broken. I mean it's still functioning, but rsync doesn't work. If I do 'ls -l /sharedfolders/NAS1' the result is 'total 0'. And so rsync can't see the folder either. If I restore OMV5 from a backup (clonezilla live) rsync works for a short time, then breaks again. I need to get on the forum and ask if anyone has any info. I wonder if clonzilla might contribute to the problem and if I should switch to Duplicati.

@DBTechYT 3 жыл бұрын

I wish I had an answer for you, but I've never used Clonezilla. In all honesty, I've only started using Duplicati recently and I'm still learning different things about it.

@jepolch 3 жыл бұрын

@@DBTechYT No problem. I was just venting. :) It only started with OMV5. Thanks!

@a19901201 3 жыл бұрын

Can it be backed up to remote FTP?

@DBTechYT 3 жыл бұрын

Yes

@gamingoutloud293 Жыл бұрын

Is a full server backup possible? not only some specific folders.

@DBTechYT Жыл бұрын

Possibly, but I can't guarantee that it will have a good outcome

@gamingoutloud293 Жыл бұрын

@@DBTechYT ok thank you for the reply. Sadly i can't find a good solution of full incremental backups. Clonezilla takes too many manual steps.

@DBTechYT Жыл бұрын

I would consider moving to something like Proxmox and a Proxmox backup server then

@gamingoutloud293 Жыл бұрын

@@DBTechYT Yes, that makes sense. But i already tried it. Proxmox / unraid has still problems to passthrough integrated graphicscard. Which i need for transcoding.

@ronald0122 3 жыл бұрын

How can you backup a complete portainer + docker environment without using omv. I use linux (ubuntu server) and i need all the needed files on restore. Now i only backup my appdata folder. In that folder i have all conf/appdata files for all containers. What for instance i only backup the appdata/conf files and i setup a new ubuntu machine and the image version differs? All my conf files are in /docker/appdata/* For example unifi controller is located in /docker/appdata/unifi I created a backup in duplicati for /docker/appdata. This works great but when my ubuntu server fully crashes i would need to setup a new portainer with new images, etc.

@DBTechYT 3 жыл бұрын

That's the general process when a server crashes: Setup your server and applications and then re-deploy your backed up data over what you just did. I suppose you could find a different solution that would take an image of your disk drive, but I'm not sure what the options are for that.

@ronald0122 3 жыл бұрын

@@DBTechYT but when i backup just the conf files thats all i need right?

@DBTechYT 3 жыл бұрын

Watch the next video where I show how to restore a container.

@ronald0122 3 жыл бұрын

@@DBTechYT thanks will do.

@MobileShophu 3 жыл бұрын

Next time please show where you put the text in portainer! You cut this moment! I know at new stack but if see a new kid it doesnt know. Thx

@DBTechYT 3 жыл бұрын

I'm pretty sure I said "In a stack"

@Shaq2k 2 ай бұрын

Too shallow really. How about stopping other containers before backing them up? And backup of containerized database servers?

@DBTechYT 2 ай бұрын

So this solution isn't for you. That's fine. There are other options out there. This was just one of them that LOTS of others HAVE found useful. I hope you find the solution that works for your needs. All the best.

@spicyF1 3 жыл бұрын

Im not a fan of this backup strategy. What I have is a Proxmox Debian Container configured with a NFS server, which is then converted into a Template. I run multiple clones of these templates to host the Docker data files. Proxmox Backup Server (Beta) does an Incremental Backup of those containers as per the schedule which is more efficient as opposed to archiving a folder. Another benefit of this as im running Proxmox on ZFS I can fine tune the block size of each disk in each container for optimum performance and compression. I can easily do snapshots, and of course restoring. All from a single system so it feels more cohesive

@DBTechYT 3 жыл бұрын

Okay. This isn't meant for people using a virtualization system like ProxMox. This video doesn't apply to you in this case. Do it your way. I'm talking to people who are running Docker on bare metal.