I have followed disk storage technology as part of my job for decades now.
I have used RAID, ZFS, BTRFS to maintain data integrity and combine multiple drives into one drive.
The problem I have for my home computer is space and power is a big issue. I don't like raid 1 because you have to double the amount of drives. I don't like RAID 5 because of the risk of data loss (yes seriously). I could go on.
I quite like ZFS but the restricted ability to add new drives to arrays makes it a no go. Unraid won't run on an existing linux system, ideal for NAS drives but not local storage.
I have watched various pooling technologies come and go, but have only just recently tripped over MergerFS and SnapRaid.
SnapRaid is not a true RAID system but uses some of it's technologies to provide you with a backup service and file validation. MergerFS allows you to merge multiple hard drives of different sizes onto a single mount point.
In combination this is a winning set up in my view.
SnapRaid provides parity data and checksums. So if a drive fails it can be used to rebuild a replacement drive. It also keeps a checksum of each file so if a file gets corrupted on disk you get notified (I used to use BTRFS for this).
It works at the file level so doesn't do drive striping and doesn't touch your data drives at all. You can put all your existing drives in to a snapraid file system along with a new parity drive and you won't lose any data during the build/sync process.
Other systems require the drives to be formatted and configured. You can literally add a new drive (with or without data on it) to the system, modify one line in the config and re-sync and the new drive is part of the array and protected.
You can pull down the array at anytime and stop using it and not lose any data. Snapraid does not touch the data drives (except on recovery). So the files stored are in their native format on their native file system.
SnapRaid is not a raid system though, it does not operate in real time. You have to periodically run a sync command. Raid is not a backup, Snapraid is not raid. Snapraid is like a backup system for local use that protects your data and allows you to undelete stuff as well.
MergerFS allows you to combine multiple drives into one mount point. For windows users each new drive needs a new drive letter. On linux each drive gets mounted to a folder. So I can add a second drive and mount it to /documents/drive2 or any other path.
The whole file system looks like one. But If I have /media and want to increase the size then a new drive would be /media/drive2 for example meaning I couldn't keep my films/music/tv etc in an organised state.
MergerFS allows you to merge multiple hard drives (different sizes) into one mount point. So your original /media folder could be added to by merging /mnt/disk2 and /mnt/disk3 and all 3 drives would appear as one drive under /media
Combining SnapRaid with MergerFS. Means you can lob that 2TB, 4TB, and 6TB hard drive into your system add 1 additional 6TB hard drive for parity and configure it with snapraid to protect the data and mergerFS to combine the 3 data disks into a single 12TB hard drive.
I have been reading up on this over the last 10 days or so and am finally convinced this is the way forward for me. I just thought I would share as it is a pretty elegant solution which wasn't around last time I looked at my personal storage situation 3-4 years ago.
github.com
PS: This is not for beginners though. You do need to understand Linux filesystem and be able to build software from scratch (mergerfs requires a configure/build/build install).
I have used RAID, ZFS, BTRFS to maintain data integrity and combine multiple drives into one drive.
The problem I have for my home computer is space and power is a big issue. I don't like raid 1 because you have to double the amount of drives. I don't like RAID 5 because of the risk of data loss (yes seriously). I could go on.
I quite like ZFS but the restricted ability to add new drives to arrays makes it a no go. Unraid won't run on an existing linux system, ideal for NAS drives but not local storage.
I have watched various pooling technologies come and go, but have only just recently tripped over MergerFS and SnapRaid.
SnapRaid is not a true RAID system but uses some of it's technologies to provide you with a backup service and file validation. MergerFS allows you to merge multiple hard drives of different sizes onto a single mount point.
In combination this is a winning set up in my view.
SnapRaid provides parity data and checksums. So if a drive fails it can be used to rebuild a replacement drive. It also keeps a checksum of each file so if a file gets corrupted on disk you get notified (I used to use BTRFS for this).
It works at the file level so doesn't do drive striping and doesn't touch your data drives at all. You can put all your existing drives in to a snapraid file system along with a new parity drive and you won't lose any data during the build/sync process.
Other systems require the drives to be formatted and configured. You can literally add a new drive (with or without data on it) to the system, modify one line in the config and re-sync and the new drive is part of the array and protected.
You can pull down the array at anytime and stop using it and not lose any data. Snapraid does not touch the data drives (except on recovery). So the files stored are in their native format on their native file system.
SnapRaid is not a raid system though, it does not operate in real time. You have to periodically run a sync command. Raid is not a backup, Snapraid is not raid. Snapraid is like a backup system for local use that protects your data and allows you to undelete stuff as well.
MergerFS allows you to combine multiple drives into one mount point. For windows users each new drive needs a new drive letter. On linux each drive gets mounted to a folder. So I can add a second drive and mount it to /documents/drive2 or any other path.
The whole file system looks like one. But If I have /media and want to increase the size then a new drive would be /media/drive2 for example meaning I couldn't keep my films/music/tv etc in an organised state.
MergerFS allows you to merge multiple hard drives (different sizes) into one mount point. So your original /media folder could be added to by merging /mnt/disk2 and /mnt/disk3 and all 3 drives would appear as one drive under /media
Combining SnapRaid with MergerFS. Means you can lob that 2TB, 4TB, and 6TB hard drive into your system add 1 additional 6TB hard drive for parity and configure it with snapraid to protect the data and mergerFS to combine the 3 data disks into a single 12TB hard drive.
I have been reading up on this over the last 10 days or so and am finally convinced this is the way forward for me. I just thought I would share as it is a pretty elegant solution which wasn't around last time I looked at my personal storage situation 3-4 years ago.
SnapRAID
A backup program for disk arrays. It stores parity information of your data and it recovers from up to six disk failures
www.snapraid.it
GitHub - trapexit/mergerfs: a featureful union filesystem
a featureful union filesystem. Contribute to trapexit/mergerfs development by creating an account on GitHub.
github.com
PS: This is not for beginners though. You do need to understand Linux filesystem and be able to build software from scratch (mergerfs requires a configure/build/build install).
