datahoarder

6758 readers

2 users here now

Who are we?

We are digital librarians. Among us are represented the various reasons to keep data -- legal requirements, competitive requirements, uncertainty of permanence of cloud services, distaste for transmitting your data externally (e.g. government or corporate espionage), cultural and familial archivists, internet collapse preppers, and people who do it themselves so they're sure it's done right. Everyone has their reasons for curating the data they have decided to keep (either forever or For A Damn Long Time). Along the way we have sought out like-minded individuals to exchange strategies, war stories, and cautionary tales of failures.

We are one. We are legion. And we're trying really hard not to forget.

-- 5-4-3-2-1-bang from this thread

founded 4 years ago

MODERATORS

archivist@lemmy.ml

What's an elegant way of automatically backing up the contents of a large drive to multiple smaller drives that add up to the capacity of the large drive? (on Linux) (lemmy.ml)

submitted 11 months ago* (last edited 11 months ago) by HiddenLayer5@lemmy.ml to c/datahoarder@lemmy.ml

8 comments fedilink hide all child comments

So I have a nearly full 4 TB hard drive in my server that I want to make an offline backup of. However, the only spare hard drives I have are a few 500 GB and 1 TB ones, so the entire contents will not fit all at once, but I do have enough total space for it. I also only have one USB hard drive dock so I can only plug in one hard drive at a time, and in any case I don't want to do any sort of RAID 0 or striping because the hard drives are old and I don't want a single one of them failing to make the entire backup unrecoverable.

I could just play digital Tetris and just manually copy over individual directories to each smaller drive until they fill up while mentally keeping track of which directories still need to be copied when I change drives, but I'm hoping for a more automatic and less error prone way. Ideally, I'd want something that can automatically begin copying the entire contents of a given drive or directory to a drive that isn't big enough to fit everything, automatically round down to the last file that will fit in its entirety (I don't want to split files between drives), and then wait for me to unplug the first drive and plug in another drive and specify a new mount point before continuing to copy the remaining files, using as many drives as necessary to copy everything.

Does anyone know of something that can accomplish all of this on a Linux system?

you are viewing a single comment's thread
view the rest of the comments

[–] cerement@slrpnk.net 2 points 11 months ago (1 children)

sounds like your main limitation is attaching the drives – if you can attach them all to a single system (ex. a separate computer or a NAS case) then at least it becomes somewhat easier to access them all at once

I was thinking JBOD but Wikipedia points out the same issue you mention with RAID 0, failure of one drive can mess up the logical volume which leads to a whole host of new issues to deal with during recovery

[–] ColdCreasent@lemmy.ca 2 points 11 months ago* (last edited 11 months ago)

Not that big of a deal when it is a backup. Raid is not a backup solution, it is a 24/7 uptime solution. If the main drive dies with JBOD, then you have the backup. If a backup drive fails, then you still have the main. Trick is to ensure any drive issues are dealt with immediately and no backup runs if there is a smart error or similar on any drive.

So having software that monitors drive health and email/notifies you is necessary.

Secondary benefit of JBOD is all drives in a pool are still readable separately.