I do an s3 sync every five minutes of my important files to a versioned bucket in AWS, with S3-IA and glacier instant retrieval policies, depending on directory. This also doubles as my Dropbox replacement, and I use S3 explorer to view/sync from my phone.
Selfhosted
A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.
Rules:
-
Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.
-
No spam posting.
-
Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.
-
Don't duplicate the full text of your blog or github here. Just post the link for folks to click.
-
Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).
-
No trolling.
Resources:
- selfh.st Newsletter and index of selfhosted software and apps
- awesome-selfhosted software
- awesome-sysadmin resources
- Self-Hosted Podcast from Jupiter Broadcasting
Any issues on the community? Report it using the report flag.
Questions? DM the mods!
I perform a backup once a week from my main desktop to a HDD, then once a month I copy important data/files from all nodes (proxmox, rpi's and main desktop) to 2 "cold" unplugged HDD that's the only time I connect them. I do all of that using rsync
with backup.sh and coldbackup.sh
I use syncthing for notes across mobile/desktop/notebook, for that and other important files the backup goes to Google Drive or MEGA (besides the offline backup).
I want to try S3 Glacier since is cheaper for cloud backup... has anyone tried?
I want to try S3 Glacier since is cheaper for cloud backup... has anyone tried
tl;dr it's too expensive for what it is (cold storage), retrieval fees are painful, and you can often find hot storage for a similar price or cheaper.
The fees to restore data make it cost prohibive to have disaster recovery runs (where you pretend that a disaster has happened and you have to restore from backup) and we all know that if you don't test your backups, you don't actually have backups.
Restores are also slow - it takes several hours from when you request a download until you're actually able to download it, unless you pay more for an expedited restore or instant retrieval storage. This is because the data is physically stored on tapes in cold storage, and AWS staff have to physically locate the right tapes and load them.
Glacier also isn't even that cheap? It looks like it's around $4/TB/month, whereas $5/TB/month is a very common price for hot storage and plenty of providers have plans around that price point. I wouldn't pay more than that. If you need to store a lot of data, a storage VPS, Hetzner storage box, Hetzner auction server, etc would be cheaper. You can easily get hot storage for less than $3/TB/month if you look around :)
I use a combination of technologies.
I keep most of my documents in sync between all my computers with SyncThing. It’s not a true backup solution, but it protects me from a drive failing in my desktop or someone stealing my laptop.
My entire drive gets backed up locally to a external hard drive using Borg. That provides me with the ability to go back in time and backs up all of my large files such as family photos and home videos.
Important documents get cloud backup with Restic to BackBlaze B2. Unfortunately, I don’t want to pay for the storage capacity to save all of my photos and videos, so those are a little less protected than they should be, but B2 gives me the peace of mind that my documents will survive a regional disaster like flooding or fire.
I use both Borg and Restic because I started with Borg many years ago and didn’t want to lose all of my backup history, but can’t use it with B2. I used to use one of the unlimited cloud single-computer solutions like Mozy or Carbonite but have multiple computers and their software was buggy, then they increased the price significantly. When I switched to B2, I found Restic worked well with it. I think they’re both solid solutions, but the way Restic works and the commands make more sense to me.
I have a lot of photos that I take. Amazon Photos gives me unlimited storage to back them all up, but it’s terrible. When Amazon Drive existed, I could grab a folder and drop it in the Photos area of Drive. My folder structure was maintained and it was easy to see what I’d already backed up or what else needed to be sent. Then Drive was discontinued and the only way to manage my photos is through the terrible web interface. There is no folder structure, putting photos in albums is unwieldy, and I have no confidence in the systems ability to give me back my photos if I needed to recover from data loss. Uploading a bunch of photos through the web page is slow and fails more often than not, leaving me to painstakingly figure out what went and what failed or just upload the whole thing again, creating duplicates. Most of the time, I can’t even find a photo or album I’m searching for. I hate that it exists and would fill a specific need if it wouldn’t have such a terrible interface.
I wish I’d have a friend who would share a few TB of storage with me but I’m pretty happy with my system, even though it has some gaps.
I've finally settled on Duplicacy. I've tried several CLI tools, messed with and love rclone, tried the other GUI backup tools, circled back to Duplicacy.
I run a weekly app data backup of my unRAID docker containers, which is stored on an external SSD attacked via USB to the server. The day after that runs duplicacy does a backup of that folder to Backblaze B2. My Immich library is backed up nightly and also sent to B2 via Duplicacy. Currently, those are the only bits of critical data on the server. I will add more as I finalize a client backup for the Win10, Linux, and MacOS devices at home, but it will follow this trend.
I backup my ESXi VMs and NAS file shares to local server storage using an encrypted Veeam job and have a copy job to a local NAS with iSCSI storage presented.
From there I have another host VM accessing that same iSCSI share uploading the encrypted backup to Backblaze. Unlimited "local" storage for $70\y? Yes please! (iSCSI appears local to Backblaze. They know and have already started they don't care.)
I'm backing up about 4TB to them currently using this method.
Mine is kind of similar. Hyper-V backed up with Veeam to a separate logical disk (same RAID array, different HDD's). Veeam backups are replicated to iDrive with rsync.
I need to readjust my replication schedule to prioritize the critical backups because my upload speed isn't fast enough to do a full replication that often.
I have a cheap 2 bay synology NAS that acts solely as a backup server for my main NAS in an offsite location as well as a USB drive locally.
Backups run every night with duplicacy
I exclude media files (movies, TV shows,...) from my backup routine due to the sheer amounts of data accumulated over time and the fact that most of it can be re-aquired using public sources in case disaster recovery is needed
Everything to Crashplan.
Critical data also goes to Tarsnap.
The main storage is a Nas that is mounted in read only most of the time and has two drives in raid mirror. Plus rclone to push a remote and client side encrypted backup to backblaze.
Cheap second NAS that I power up every now and again, then I run a dsynchronize profile which replicates the important stuff (video), and all the stuff I could never replace I put on a usb and keep it elsewhere
I have my data backed up locally on an HDD, though I'm planning on building a server machine to hold more data with parity (not just for backups). Important data I have backed up in Google drive and Proton drive, both encrypted before upload. It isn't that big, I don't back up media or anything in the cloud. Oh and I have some stuff in mega, but I stopped adding to that years ago. I should probably delete that account, thanks for the reminder!
Backend storage is all ZFS. I have a big external drive plugged in via USB on my ZFS box and that backs up my daily backups.
I have a two old PCs that I run ZFS on as well. One auto turns on every week and ZFS backs up to that. The other PC is completely manual and I just randomly turn that on and backup. Every so often. Usually every 2-4 weeks.
For off-site backups. I use Syncthing and it is running on a server at a families house. Few miles away.
I picked Syncthing over ZFS because I actually a little more than an off-site I wanted a two way sync between our two locations so both locations could have a local copy they can edit and change.
I still need to get it set up, but I'll have 3: One on my NAS, one on a local USB drive, and one offline backup. I'll use rsync for the job.
Backblaze. Easy and cheap. It’s fire and forget for the most part.
My work is using Google drive for Sync/back up so that is covered by them.
Personal data is automatically synched (syncthing) between three computers in different rooms in my home + some of the files is copied to my phone and tablet. I consider adding also an online server for further redundancy
Local versioning with btrfs rsync copy to other machine in home network rsync to NAS at my parents home
rsync over ssh (my server is in the next room) which puts the backup on an internal drive. I also have an inotify watch to zap a copy from there to an external USB drive.
Nightly backups to an on-prem NAS. Then an rsync to a second off-site NAS at my folks house.
Encrypted files sent to Google Cloud Storage (bucket) for long-term archival. Comes out pretty cheap like that.
I have a compressed copy of the config files on my server on a separate drive, and every night restic makes a snapshot and stores it in a separate drive attached to a raspberry pi 3.
Usb drive
For a long time I did 1 hot copy (e.g. on my laptop), 1 LAN/homelab copy (e.g. Syncthing on a VM), and 1 cloud copy ... less a backup scheme than a redundancy scheme, albeit with file versioning turned on on the homelab copy so I could be protected from oopsies.
I'm finally teaching myself duplicity
in order to set up a backup system for a webdev business I'm working on ... it ain't bad.