This is NOT a knock against the author, I just wanted to point out that "backups" are much more complicated than "copy files elsewhere". For DB's I'd probably consider running a replica on 1 or more other clouds. IDK the logistics of replication over the internet but I know for work we do replication from our datacenter down to our local servers and that's over a relatively slow connection so I assume it's possible to do it from cloud-to-cloud.
In that case, it does become as simple as just copying files elsewhere. (For example, using the Restic steps in my post to backup a folder of hourly database dumps, like you mentioned.) Replicating databases (and other methods made specifically for DBs) is certainly a much, much better route for mission-critical and/or enterprise data.
Covering every permutation of different types of data to backup would have made a long post much longer, but I'll add writing a part two to my to-do list covering rudimentary database backups since that has been brought up here a few times.
Thanks for the feedback! :)
I am definitively looking at this through the lens of where I work where a mysqldump (or equivalent) could take days to complete in full (DB is nearing 2TB in size now). For a number of projects a mysqldump might only take seconds or minutes and would be a perfect candidate for this backup scheme.
Replica's aren't actually a backup, though it's probably a good idea to have them, and in the case of too much data to reasonably backup, as close as you are going to come.
Also, depending on data structures, for example if your data models fit into something like MongoDB or not, it's easy enough to trigger a dump for each record. I've setup systems where primary records are json+gz files in S3 or Azure blobs. In practice this has been as part of the process that will update ElasticSearch (or Mongo) from an RDBMS authority.
It was pretty easy to do <base>/collection/{ID}.json.gz .. from there, worst case, I'd have to create a re-population script from the hard files, and might lose a little less important data, but would always have a hard recovery path. YMMV of course.
I've got 3 side projects that I've been muddling on getting started and some of the things that have happened to the original author frankly scare me. I'm also working on legal structure (LLCs etc) before any real launch. There are so many edge cases to consider, and sometimes it's hard just getting started and accepting that you will make mistakes along the way.
For Postgres you can use something like WAL-E to continuously backup to S3 or other cloud storage providers. The underlying mechanism is explained in the Postgres documentation as "Continuous Archiving and Point-in-Time Recovery (PITR)"(https://www.postgresql.org/docs/11/continuous-archiving.html). Using this method you lose a minimal amount of live data when your main server goes down. And if you need to restore, you can also restore at an arbitrary point of time, so e.g. just before you accidentally deleted everything in a table or something like that.
It's not the topic of the post, but database backups deserve a special mention. You can't just naively copy the database folder this way in most cases, you have to make sure to backup a consistent snapshot of the database. This is still not hard to do at smaller scales, when you can just add an exported dump of the database to your regular backup. But it is a point that needs some attention if you host the database yourself.
And do rehearsal as well. Backup may not work.
But yes, you do have to test and verify that your backup works. It might be configured entirely wrong, the cron job might not be running for some reason, you set it up with encryption years ago but lost the passphrase. There are plently of ways this can potentially fail.
A better way would be to unmount and send the filesystem with 'dd' or something like that, or, to use 'zfs send' but I have a suspicion that neither of those options are available to you ...
I will say that splitting the rsync job (rsnapshot runs rsync underneath) into multiple, smaller jobs, could save you some time if you're running into any resource limits while you walk that big set of inodes... so if you're lucky and you have 4 or 5 or 8 top level dirs that are all roughly the same size, you could do a handful of smaller jobs, one after the other, instead of one huge one ...
To add to that.. to avoid having to unmount your filesystem; use LVM. Then you can call `sync`, snapshot your main volume, and `dd` the clean snapshot. Once you're done, remove the snapshot. This strategy avoids downtime while backing up your volume.
The issue is with how network failures (which always eventually happen) interact with the "uninterruptible" Linux process state. Hell breaks loose, and the failure is anything but obvious.
What I don't like about duplicity is how it spews weird error messages that I gathered were related to Python versions. It was easier to just start over with restic and haven't had any problems since.
A conclusion from one comparison (2017)[2]:
"Restic’s memory requirements makes it unsuitable for backing up a small VPS with limited RAM, and the slow backup verification process makes it impractical on larger servers. But if you are backing up desktop or laptop computers then this may not matter so much, and using Restic means that you don’t have to setup your own storage server."
Is this still true?
I'm still convinced that its dedupe is magical. I don't know if there's a backup app that is more frugal with disk space, but Borgbackup has served me well in a non-growing 1.5TB backup area for 3+ years now.
I switched to restic now, which allows to take incremental backups, but can also remove any snapshot to prune the history. Although it does not support compression, due to its deduplication and removing the need to store multiple full backups, the restic repository takes less space now than duplicity before.
But as you point out, if you don't need a long history, incremental eventually gets more expensive. Unless you could squash older than X, I suppose, but presumably that's so expensive to run (encryption & compression) that it's not supported.
In theory, you need to be ready to literally delete every container at any time and pull them from scratch and be 100% fine, since all of your actual data should be stored on the host and mounted as Docker volumes [0]. It's a good Doomsday test if you're looking for one. ;)
With kubernetes (and no more specifics than you've mentioned) you should use a Job.
With docker-compose I think I'd be tempted to have a different service that isn't long-running, and a cron job on the host that runs it.
With swarm, unless it supports something like k8s Jobs, (I don't know if it does or not, only used it once briefly and in anger) I'd probably have a 'cronjob' service which was responsible for launching the short-lived services per compose suggestion above.
Currently just doing a combination of s3cmd for a local archive (takes forever to download and then it doesnt seem like incremental syncs are any faster), as well as having Google Console clone my bucket there (but I'm not sure if it's versioned, or as easy as downloading the whole archive).
Never used duplicity -- would it be fast for something like this? Guessing I should just cron it on a remote server instead of running off a local machine frequently.
For maximum peace of mind, always pull backups from a separate server that is not exposed to the world. Don't let your primary server push arbitrary data to the backup store.
This rule is trickier to follow when your backup store can't run scripts, which is why so many tools designed to work with S3 tell you to keep the keys exposed. But if you really want to, you can use an intermediate host to pull backups before pushing them again to S3.
[1] https://borgbackup.readthedocs.io/en/stable/usage/notes.html...
I know that tarsnap [1] can work with `list`, `write` and `delete` keys.
Combine that w/ S3 Object Versioning and you’ve got a pretty solid approach.
It depends on what you want to do, but can probably be accomplished in a <= $5 VPS.
EDIT: Oh and restic has much more functionality than rsync, including deduplication and encryption. rclone is more of a "glorified rsync", but even then its array of backends makes it truly glorious.
[0] https://wiki.archlinux.org/index.php/Synchronization_and_bac...
https://www.rsync.net/products/restic.html
One of the modes of restic is SFTP target and as we run stock, standard OpenSSH, it works perfectly.
EDIT: A sibling comment to yours mentioned 'rclone' and I am happy to informally announce that over the past few months we have rolled out the 'rclone' binary to all of our production fileservers (it requires a server-side binary exe to be in place) and it is being used by rsync.net customers to broker file transfers cloud to cloud to cloud (as rclone is apt to be used for). 'rclone serve' and 'rclone mount' are disallowed for (I think) obvious reasons, but otherwise everything works ...
I have a bunch of snapshots (https://github.com/jakelee8/zfs-auto-snapshot) that I want to backup along with the active tree. But don't want to keep extra copies of the data.
- Do these services offer snapshotting? ...that can be automated?
- Is there zfs integration, e.g `zpool send | b2 receive`?
1) Upload a custom ISO with ZFS (https://github.com/beren12/zfs-iso/)
2) Create a new VPS without OS and boot to your uploaded ISO.
3) Create a ZFS root pool and bootstrap your Debian or another distribution.
4) Enable all cool features: compression, encryption, etc.
5) rsync your zfs snapshots from Vultr to Linode and vice-versa.
This is how I do. You can even use them as templates for newly VPS.
And for backups, BackBlaze B2 and WASABI with a zfs-snapshot-upload script.
Just found a good comparison/benchmark of the two at [1] - tl;dr seems to be that restic is fast, and duplicity is small.
Each VPS backs up to the other. RSnapshot makes daily diffs that use hardlinks to avoid taking up space. This also mitigates tampering, as only root have access to the snapshots.
Demo site using anon login for testing: [1]
The technical side could be hidden from less technical users and it sold as isolated servers so the data would be protected.
I don’t have the skills or the time to work on this so happy for others to use the idea
(I'm just a customer that has been generally pleased over the past many years)
