Unfortunately, as Lemmy doesn’t appear to have an indexed search function I am not sure if this is a common question or not. Please point me to the necessary thread/s if this is answered elsewhere.
I’m looking to continue the ball rolling on my home server. Jellyfin setup was a nice dive back into networking, which I haven’t done for quite some time and the logical next step is to get all the data we want to retain into a single hub.
Most searches seem to point to syncthing with nextcloud, but before I get started, I want to check I am really going in the right direction.
I would like to primarily remove the space burden from my devices and dump them all onto a few drives and a cloud backup (in case of physical loss of drives). I believe syncthing can do this, but some appear to say that it is not an effective archive tool.
I would like to be able to retrieve this data without much hassle for e.g. photo editing, and place the finished file back on the server. Preferably the local copy would be removed again, to reduce the need for extra space on each device. I would like to run this over nextcloud, but might be misunderstanding the software a bit.
As others have said, sync isn’t backup.
It may be part of a backup plan, however.
I use Syncthing on my mobile devices to keep data created on the devices synchronized to my server at home. Things like photos sync to home over any connection, while I sync other stuff only over wifi. Syncthing-Fork allows you to set these conditions on a per-folder-pair basis.
That server becomes my authoritative box for any data. All that data is then mirrored on a schedule to 2 other systems at home (a NAS and a large drive on another box).
The main server also has a cloud backup which runs continuously.
So I have 3 local copies of data to recover from if I have a hardware failure, and a cloud backup.
I find tools like Syncthing and Resilio are good for synchronization, especially mobile devices. But between full-pc-OS devices, I just use native tools (scripts and schedules) because I don’t want synchronization, but specific patterns of copying/mirroring, etc.
I do use Resilio for ad-hoc access to almost any file on my server, since it’s Conditional Sync feature permits me to connect with a mobile device from anywhere and sync only the selected files. So I can grab a movie or TV show, Resilio will sync it and I can watch it once the sync is complete.
Syncing software is not a backup. I’ve had cases where they get confused and end up deleting data. They’ll also blindly copy over corrupted or randomwared files.
Randomware should be a thing!
It was, shareware. 10000 games on this CD!!!
You’ve not made it clear what exactly it is you want. Nextcloud or syncthing are good for syncing personal files. If you want to make server backups, they’re not gonna be the way.
If you want to automate backups, you could just use a cronjob to make a tarball and rsync it?
Better yet, use borg to back up. Managing your own tars is a burden. Borg does duduplication, encryption, compression, and incrementals. It’s as easy to use as rsync but it’s a proper backup tool, rather than a syncing tool.
Not the only option, but it’s open source, and a lot of hosts support it directly. Also works great for local backups to external media. Check out Vorta if you want a GUI.
Remember sync isn’t a good backup. You’re thinking of loss of drives but if this is important data you need to also consider mistakes.
If you accidentally delete files you shouldn’t, you don’t want this deletion to sync to all your copies so it’s gone for good and the backup doesn’t help.
Personally I use borgmatic to keep incremental, deduplicated backups. Then I can go back to previous states.
If you install nextcloud all in one, it comes with a backup solution (also borg based). Then devices don’t need a copy of every file. But you’ll want your server to have a backup drive for this.
I then sync my borg backup to a backblaze b2 bucket for offsite, encrypted backup using rclone. That then meets the 3 2 1 backup plan.
I notice you mention Jellyfin. I don’t back up my Jellyfin media, the cloud storage for that could get very expensive and I could get it again if I needed it.
If you accidentally delete files you shouldn’t, you don’t want this deletion to sync to all your copies so it’s gone for good and the backup doesn’t help.
edit folder > ignore delete
and you don’t have to worry about syncthing deleting your backup
Yip you can do that but then it’s messy! And what if you overwrite a file by accident?
And if you do lose your hard drive then you have a weird state to restore from.
I’d much prefer the ability to restore to a point in time that comes with something like borg.
And what if you overwrite a file by accident?
you can activate one of the options in the “file versioning” to keep deleted files, up to x versions and however long you want
And if you do lose your hard drive then you have a weird state to restore from.
are you writing about losing the backUp drive?
I think versioning is the better option.
are you writing about losing the backUp drive?
No, losing your main version. Imagine you have a computer with syncthing and a server where it syncs to. If you chose no deletions, then it will sync all files to the server but all the stuff you deleted (draft documents, random files, photos from that time your kid held the camera button on your phone down and took 3000 photos in 30 seconds) will be deleted from your computer but still there on your server.
When you computer gets struck by lightning and everything is destroyed but the server is fine, now you have to re-sort out all your files because all the stuff you deleted is still on the server version.
Your suggestion of enabling the option to keep previous versions is probably cleaner. Personally I prefer to keep previous versions and deduplicate to save space.
That, on the other hand, is only viable, if you are sure, data never needs to expire. Dedicated backup solutions work with retention policies.
Just to add to this, if you have periodic snapshots on the server side, this does solve the problem. And it simplifies things a lot.
Yes, if you go with something like syncthing, have it also sync to a server where you run borg backup so you get the incremental backup.
Lemmy definitely has a search function. Due to federation (and specifically how lemmy.zip has federated with lemmy.world) you can easily search from your home instance of lemmy.zip and limit your search to this community
https://lemmy.blahaj.zone/pictrs/image/fc1f7747-1258-471d-8db0-4f0bbc77e7ce.webp
I haven’t tried syncthing yet, but might be able to combine it with some of these ideas.
I’m currently doing phone backups to a samba smb server with an app called smbsync2 from fdroid. It can copy or move files and directories from and to a smb share on a schedule. An option for desktop would be rsync.
To get the archiving functionality you can do automatic ZFS snapshots. You can restore the entire snapshot or mount it via the terminal, or samba apparenly has a feature to display files from snapshots as shadow copies on windows.
There is one limitation of this is, if a client deletes a file on the local storage, the file on the server doesn’t get deleted automaticly.
I second smbsync2 for phone backup. It’s very customizable and once setup, just works.
I also use syncthing but only for syncing files I want to be the same on all devices. FreeTube watch history and subscriptions, stuff like that.
One advantage of Syncthing over smbsync is it can use any network connection, not just LAN.
You can configure to only use LAN, or even specific LANs if needed (identified by SSID). I have a few sync jobs that work this way.
Off to test smbsync now - thanks for the recommendation!
Adding to what everyone else has already said, you want sync and backup.
Sync to a central location and backup from there.
For sync, you want syncthing or nextcloud. I would lean towards syncthing for media. If you had a million files in a complex folder structure and a dozen users with different access requirements and instant sync and collision protection is important then nextcloud might be the go. Otherwise syncthing is much more manageable.
My recommendation with syncthing, which is not obvious, is to set up a single hub which each client syncs with. By default you end up with a mesh where everything is connected to everything. It’s very difficult to manage with a lot of folders and devices. Turn off discovery and input the server / hub details manually.
For backup, if you have a lot of media you want deduplication. If yesterday’s backup included ABC and today’s is ABCD you only want to transfer D. This is similar to an incremental backup, but the subtle difference is that with deduplication the most recent backup is the “full backup” with the “diffs” going backwards in time, allowing you to purge old backups. I like borgmatic but there are others.
I would also consider carefully exactly what is worth backing up on what service. I don’t backup movies and tv series at all.
My final recommendation is, it’s critically important to test deploying your backups regularly.
3 copies of your data (original and 2 backups)
2 different media. (pick 2: Raid, jbod, ssd, tape, managed cloud)
1 off-site
I’m gonna add in: one offline
So
original data on a mirror raid or raid 5
External disc disconnected backup trued up whenever you can
Syncthing/rsync/b2/S3 whatevs to get it off-site.
The nice part about syncthing is it will stop if a certain percentage of you data changes. It has a good chance to stop ransomware as long as it’s not too slow at the encryption.
Exactly what are you trying to back up? If it’s for something like, let’s say, a password manager, syncing is the worst way imaginable.
But if it’s for your Jellyfin media backup, rsync is pretty sufficient imo. They rarely change, costs pretty penny to back up and (relatively) easy to rebuild.
So it all comes down to what your scenario is.
