Title is TLDR. More info about what I'm trying to do below.
My daily driver computer is Laptop with an SSD. No possibility to expand.
So for storage of lots n lots of files, I have an old, low resource Desktop with a bunch of HDDs plugged in (mostly via USB).
I can access Desktop files via SSH/SFTP on the LAN. But it can be quite slow.
And sometimes (not too often; this isn't a main requirement) I take Laptop to use elsewhere. I do not plan to make Desktop available outside the network so I need to have a copy of required files on Laptop.
Therefor, sometimes I like to move the remote files from Desktop to Laptop to work on them. To make a sort of local cache. This could be individual files or directory trees.
But then I have a mess of duplication. Sometimes I forget to put the files back.
Seems like Laptop could be a lot more clever than I am and help with this. Like could it always fetch a remote file which is being edited and save it locally?
Is there any way to have Laptop fetch files, information about file trees, etc, located on Desktop when needed and smartly put them back after editing?
Or even keep some stuff around. Like lists of files, attributes, thumbnails etc. Even browsing the directory tree on Desktop can be slow sometimes.
I am not sure what this would be called.
Ideas and tools I am already comfortable with:
-
rsync is the most obvious foundation to work from but I am not sure exactly what would be the best configuration and how to manage it.
-
luckybackup is my favorite rsync GUI front end; it lets you save profiles, jobs etc which is sweet
-
freeFileSync is another GUI front end I've used but I am preferring lucky/rsync these days
-
I don't think git is a viable solution here because there are already git directories included, there are many non-text files, and some of the directory trees are so large that they would cause git to choke looking at all the files.
-
syncthing might work. I've been having issues with it lately but I may have gotten these ironed out.
Something a little more transparent than the above would be cool but I am not sure if that exists?
Any help appreciated even just idea on what to web search for because I am stumped even on that.
Simplify it down if you can. Forget NFS, SSHFS and syncthing as those are to complex and overkill at the moment. SMB is dead simple in a lot of ways and is hard to mess up.
I would start by figuring out how to get more drives in your desktop. SSDs can be put anywhere but HDDs need to be mounted. I would get two big disks and put them in raid 1 and then get a small disk for the boot drive. TrueNAS is still going to be the easiest route and TrueNAS scale is based on Debian under the hood although its designed to not be modified for reliability.
Forget the USB drives. I did that for a while and it is bad in so many ways. It will come back to haunt you even if the work fine for now.
For the desktop you always could upgrade the ram or CPU if that becomes the bottleneck. If the ram is 8gb or less you should upgrade it. Make sure you are getting ram that is fast enough to keep up with the CPU. You can find more information about your CPU and its ram speed online.
For your WiFi, it actually isn't bad. You might be ok using it depending on what you are doing.
For the minipc I wouldn't use it as a NAS. You can but it is probably going to be limiting.
In case your desktop doesn't have enough Sata ports you could get a ePCI sata card. They are pretty cheap ($10) and will potentially work better if your hardware is limited.
OTOH, SSHFS and syncthing are already humming along and I'm framiliar with them. Is SMB so easy or having other benefits that would make it better even though I have to start from scratch? It looks like it (and/or NFS) can be administered from
cockpitweb interface which is cool.Now that I look around I think I actually have a bit of RAM I could put in the PC. MacMini's original RAM which is DDR3L; but I read you can put it in a device that wants DDR3. So I will do that next time it's powered off.
Thanks for letting me know I could use an expansion card. I was wondering about that but the service manual didn't mention it at all and I had a hard time finding information online.
Is this the sort of thing I am looking for: SATA Card 4 Port with 4 SATA Cables, 6 Gbps SATA 3.0 Controller PCI Express Expression Card with Low Profile Bracket Support 4 SATA 3.0 Devices ($23 USD) I don't find anything cheaper than that. But there are various higher price points. Assuming none of those would be worthwhile on a crummy old computer like I have. Is there any specific RAID support I should look for?
I have only the most cursory knowledge of RAID but can tell it becomes important at some point.
But am I correct in my understanding that putting storage device in RAID decreases the total capacity? For example if I have 2x6TB in RAID, I have 6 TB of storage right?
Honestly, more than half my data is stuff I don't care too much about keeping. If I lose all the TV shows I don't cry over it. Only some of it is stuff I would care enough to buy extra hardware to back up. Those tend to be the smaller files (like documents) whereas the items taking up a lot of space (media files) are more disposable. For these ones "good enough" is "good enough".
I really appreciate your time already and anything further. But I am still wondering, to what extent is all this helping me solve my original question which is that I want to be able to edit remote files on Desktop as easily as if they were local on Laptop? Assuming i got it all configured correctly, is GIMP going to be just as happy with a giant file lots of layers, undos, etc, on the Desktop as it would be with the same file on Laptop?
Funny you found that card as I have the same one. I would only spend the extra money if your computer doesn't have enough sata ports.
For raid you do not want hardware raid. It is generally problematic and doesn't have any error correction. Also, if the controller were to die you could be in trouble.
The reason I say TrueNAS is that it makes the setup pretty straight forward. I have never used cockpit for file shares so I do not know if it works well or not. I do no it will require more setup and maintenance. For the software you could setup either a btrfs or ZFS raid 1 config. TrueNAS is ZFS only but it is turned for solid performance in a single job.
The reason I say raid 1 is not because it is a backup. Raid is not a backup and should not be treated as such. What it can offer is reduced downtime. If one drive fails you can simply swap out the drive. If you do not have a redundant copy you either need to recreate the data from a backup or try to do a data recovery operation. I would also recommend that you turn on snapshots as if it really handy to be able to roll back a change quickly. Snapshots are not complete backups but they are a form of version control.
As far as DDR3 goes, it is pretty slow and will not be able to come close to saturating a gigabit connection. Prepare to have slow speeds unfortunately. The good news is that if you get this setup and later decide it is time to upgrade then you can.