this post was submitted on 30 Dec 2024
148 points (100.0% liked)

Sysadmin

7793 readers
1 users here now

A community dedicated to the profession of IT Systems Administration

No generic Lemmy issue posts please! Posts about Lemmy belong in one of these communities:
!lemmy@lemmy.ml
!lemmyworld@lemmy.world
!lemmy_support@lemmy.ml
!support@lemmy.world

founded 2 years ago
MODERATORS
 

Went to do a test restore of one of my databases and I noticed the dump files over the last few months were all 0kb. Glad I caught it this way and not because I needed to restore. Put it on your calendar, schedule a test restore of your critical stuff a couple times a year. I know y'all are busy but it is worth the time and effort. A backup you can't actually restore isn't a backup at all.

you are viewing a single comment's thread
view the rest of the comments
[–] ComradeMiao@lemmy.dbzer0.com 2 points 4 days ago (1 children)
[–] sylver_dragon@lemmy.world 6 points 4 days ago (1 children)

It depends on the type of backup:

For a filesystem backup, restore one or more files to a secondary location. E.g. pick a few files out of the backup and try to restore them to a temporary folder. Then hash the original and restored files to verify integrity.
For a full machine backup (e.g. VM backup), restore a copy of the machine to a test location. Spin up the test machine to verify that it can boot.
For a database backup, restore a copy of the database to a test location (e.g. change the database name as part of the restore process), compare a few tables against the real database to verify integrity.

Pretty much, it's going to be some version of "Restore X to a test location and verify integrity". You want to both prove that the backup can be restored and that the restored copy is actually intact.

[–] ComradeMiao@lemmy.dbzer0.com 0 points 4 days ago (1 children)

Thanks for explaining this! Very helpful and easy to understand. Do you have preferred programs for the two actions? I currently just rsync my servers.

[–] capital@lemmy.world 2 points 4 days ago

I use restic to back up files to Wasabi.

It's all scripted but the steps are:

  1. Back up
  2. Prune based on rules
  3. Perform a repository consistency check and do further checks (takes longer) on 1% of the repo
  4. Choose 1 random file from the current backup set and get the hash of the current file
  5. Restore that file to a temporary location and hash it.
  6. Compare hashes
  7. Send push alert to me with success/failure and a summary