A data corruption bug in OpenZFS? (opens in new tab)

Sparse files make more sense if you see the file system and paging as unified. If you have allocated an array of 1 billion items, accessing the last item doesn't make the OS zero out everything from 0th to the billionth item, allocating millions of pages along the way. Virtual emory is sparse; so just one page of virtual memory is allocated. Mmap'd sparse files behave the same way.

This a feature I was completely unaware of. Why would you choose to use a sparse file instead of multiple files?

bclones were only one way to trigger the corruption. This is not a good way to check.

It's also not worth checking for because this bug has existed for many years. Your data probably wasn't affected. None of the massive ZFS storage companies out there ran into it by now either.

Your data is fine. Sleep easy.

Oracle isn't copyright holder for OpenZFS. That's one part that OpenSolaris and OpenZFS projects managed to ensure. What Oracle could do was to close OpenSolaris again under proprietary license, something that Brian Cantrill IIRC blamed on the use of copyright assignment, and that open source projects should never use it - with that as a specific example.

OpenZFS devs have openly declared that no, they are not pushing to include OpenZFS into Linux kernel, and that separate arrangement is just fine, especially since it allows different release cadence and keeps code portable.

Mainly there's an issue with certain Linux Kernel big name(s) that like to use GPL-only exports (something that has uncertain legal status) in a rather blunt way, and sometimes the reasoning is iffy.

> Just because Shuttleworth thinks he cant be sued by Oracle for including ZFS in Ubuntu with some hastily reasoned shim doesnt mean Oracle wont nonchalantly send his entire company to the graveyard just for trying.

Canonical has been shipping the kernel with ZFS for more than 7 years and so far they have not been sued by Oracle.

How is the CDDL any more detestable than the GPL family of licenses? Not saying that they are detestable in any way, but the CDDL is also a free software license so I don't get how it's worse or bad

> You'll see a stable BTRFS and a continued effort to polish XFS/LVM/MDRAID before openZFS ever makes a dent.

Right now I would put my money on bcachefs[1] rather than BTRFS. bcachefs is currently in the process of being merged into the kernel and will be in the next kernel release. Doesn't currently quite offer everything ZFS does, but it's very close and already appears more reliable than BTRFS, and once stuff like Erasure Coding is stable, it'll be more flexible than ZFS.

[1] https://bcachefs.org

Oracle can’t do a thing to an Isle of Man corporation.

Who on earth is trying to bully Linus into anything? Where have you seen that?

This is bull**! It's time for a new license to throw off the old: The Uniform Pirating License! It's a license which you stick on anything which you need a license, and it conveys all rights and zero obligations to you. Possession of the code is sufficient to run, change, and propagate code. Legal system be damned; we have cryptography and Tor, The State's law here is irrelevant (also when did you give The State license to bully you around anyway?)!

My fix: spin up a .onion to host my distribution of the Linux kernel containing ZFS integrated and BtrFS excised, do not answer abuse/legal emails, don't even have email to receive aforementioned emails. What's the pencil-necked shrimpy IP lawyer at Kernel Foundation going to do? Shut down Tor?

Any idea which diagram in PlantUML more specifically? I looked at a handful of the PlantUML categories (each one with dozens of examples) and haven't seen anything like the diagrams in OP's post.

Well, to be fair they tried and failed to reproduce the corruption that was reported. While I agree that I'm not ready to dive into bcachefs, I'm not exactly swayed by this bug report.

As a grizzled old storage admin who somehow made a career out of database backups, I wholeheartedly agree with all of this. Especially having someone else do a test restore. They don't have to be junior, just not intimately familiar with the systems involved.

One nice thing about a circa 2010 MySQL setup is setting up a new replica is easiest by restoring a backup. If you have to do that from time to time, your backups get tested by regular process.

Yes! The worst is that even if you set up all kinds of reports etc. on what you expect, if the backup runs for weeks/months successfully, you just stop paying attention and then when something fails, you won't notice it.

I do think that file systems that support snapshots - like ZFS, but I think LVM can be used for stuff like ext4, and Apple APFS does too - is the way to go. Not sure how well NTFS's Shadow Copies/Volume Shadow Service work, I heard horror stories, but not sure if those are one-off freak accidents. Probably worth considering ReFS anyway these days on a Windows Server. But with a Snapshot, you're at least insulating yourself mostly from changes to the data you're backing up. At the expensive of managing snapshots, that is, getting rid of old ones after a while because they keep taking up space.

Well data protection is expensive, nobody said the contrary.

Backup what you value the most, ignore what you don't and apply tiers depending on what needs to be kept but you can deal with transferring it back home slowly and what you need immediately in case of a failure.

My rules of thumb are:

- always invest 3x the price of your hot live NAS storage in backups. If you can't afford buying 40TB of storage, you can't afford having 10TB of live storage. Period. Goal is to have at least one copy locally and one externally and have more space to on the backup storages to account for retentions, changes and help with migrations.

- if you can't afford 3 redundant storages(RAID), favor having 3 times non redundant storage (no RAID) over having less copies of redundant one.

Additional tip to reduce cost and avoid expensive cloud offering is to find a reliable and trustable relative or friend that can host your external copy of your backup. Nebula or Tailscale now makes it very easy without having to configure routers and stuff. In exchange you can offer that person to host his/her backup storage.

Also digitalizing material stuff is nice, but printing digital photos is also a great way to preserve copies. I'd rather save the photos I cherish the most than having 3 backup copies of 10TB of blurry or non outstanding photos. After years of having them all digitally, I am inveting back in printing photos and making albums. You can also print photobook multiple times and have some stored at a relative's place.

As the sibling says, 40 TB is not exactly "average home nas" territory. What I personally do, though I don't have 40 TB available even if I counted all my hard drives together, is I just have a second device that can hold the data and back up to it regularly.

My NAS has something like 5 TB used. It's all synced to an old server that can hold about 8 TB and that's off most of the time (no fun living next to a jet engine). This cold server lives at my parents' house.

My "really important stuff" on the NAS, which is a few hundred GB of pictures and such, is regularly backed up to a bucket with object locking.

My "super important stuff", which is my company's accounting and other such documents, and lives on my laptop, is backed up to the live NAS and handled there as the really important stuff. I also back up my laptop to two normally offline external drives, one of which lives in my apartment and the other at my parents' house.

Everything non-cloud is ZFS, so after each backup to an external drive or "cold NAS", I run a scrub to make sure it is still operational. The live NAS runs a scrub every Monday morning.

Granted, this is not a "modern NAS" environment, since it made no sense to me to forego the free servers that my employer was going to send to the trash and buy some expensive off-the-shelf solution without the guarantees of ZFS (despite the issue TFA talks about). I know about power usage, but my live NAS eats less than 50W at idle (which is 99% of the time), so breaking even with the electricity prices in France would take forever.

I agree with you completely that it's used in too trite away. Which I think has echoes to backups and a lot of other "data hygiene" things in general (like doing backups at all initially, or strong passwords, or setting up new systems) which our industry has a long and unfortunate history of leaving manual and assigning a PEBKAC to when what was really needed was more automation. Manual effort doesn't scale, and cost is absolutely a critical issue for a long tale of data owners. A fundamental part of the entire value of ZFS and NAS for that matter is automating away all sorts of issues surrounding data integrity, from checksumming to disk integrity to backups, and doing so in a way that's highly dependable.

Which is how it should be. Yes bugs can happen but there's only so many 9s most of us can chase on our budgets. And "always test backups" in particular adds cost. Testing means restoring onto hardware that you can then use live, separate from your actual primary hardware or at a minimum on primary hardware with >2x the set size and enough performance to squeeze it in during downtime or around work. So yet another big increase in cost. "Testing backups" isn't trivial.

I have about that much data and LTO-6 (2.5tb per tape), and it's a huge PITA. I'm probably doing it wrong, but this is what worked for me: making an ext4 filesystem as a file, exactly 2500gb in size, formatting it, and stuffing it with data until there is < 5 gb free. take the checksum and manifest of that file, and write it to tape (takes 4 hrs without verify, plus another 1-3 hrs (can't remember now, its faster) to verify. repeat until your 40tb is done.

I know you can use ZFS snapshots but I'm not experienced enough to trust that I could make a 20-40tb snapshot without screwing something up. Plus it's all video files so I can roughly keep track of what's what and I can ignore the stupid LTO compression.

It takes days, its noisy, and very tedious. But thats #hoarderLyfe lol

“Average home NAS user” doesn’t have 40TB of data. With a subset of data that’s important like photos it’s not that expensive and with Backblaze and other services that are directly integrated in operating systems like Synology also not that hard to do.

LTO. I bought an LTO-5 system to backup 6TB of critical data and 12TB of nice-to-have data. LTO-6 is better if you can afford it.

Downside to tape backup is you need throughput, or the ability to do disk-disk backups

If you really need 40TB of irreplaceable data, then I think S3 Glacier Deep Archive might be worth looking at. According to the Amazon calculator it's something like $45/month, though of course the data might take a while to get ready if you need to restore it. There are other S3 Storage tiers as well, that are a bit more expensive but offer quicker recovery. Backblaze B2 looks like it would be about $240/month, which is IMHO also pretty reasonable for 40TB. I haven't calculated the initial traffic costs though, I assume the first upload might be a bit costly, but once it's up there, you just pay storage until you need to restore it.

If you can figure out how to split the data into categories, you could save money as well. E.g., which of this data is truly irreplaceable - stuff like personal photos, source code, whatever it is that can never be re-created. If you're running a business, then stuff that needs to be available immediately in order to keep the lights on. Those things needs to be on storage that also gets backed up daily, preferably in full, and preferably to multiple clouds.

Stuff that can be re-created from sources (e.g., rendered outputs) are less critical because in the worst case, you can just spend some days/weeks to re-create it.

Also consider regular offline backups - put it on a tape drive or on some hard disks/SSDs or even optical media (yes, it would take something like 400 BDXL disks to back up 40 TB, but I assume the data doesn't rapidly change) and put it in some offsite storage facility in case your place burns down.

My cheap solution for large datasets is to buy a raspberry pi and external hard drive(s), setup in a friend or relatives house, and setup syncthing. One friend has a copy of my ripped discs, my parents have copies of my photos, etc. Make sure the remote instance is in read only mode.

For sensitive data I would run something else that can be a Restic target so backup data is encrypted, I currently use a cloud drive that supports WebDAV for that.

I don’t try to backup my Plex library. Most of my family pictures and videos are on my MBP and I rsync the picture folder a couple times a month to the NAS. Every 6 months I get my cold storage 6TB drive and back up what I can. My MBP runs Backblaze so I have another backup of my most critical items.

AWS S3 Deep Glacier is really cheap nowadays (at least in some zones), on the order of $1/TB. As an average home NAS user with 8TB of data, I've finally taken the plunge and started backing it up. It was never worth the cost before.

I use restic to back up my NAS to Hetzner storagebox.

Also, you can probably tier your data. Maybe you don’t need same level of backup for all your 40TB.

> The last time I looked it could easily cost hundreds of dollars per month to back up as little as 40TB to the cloud.

You only have to backup the data that is important to you and you don't want to lose in case your house gets robbed, floods, burns down, etc.

If you don't mind losing 40T of data, you don't have to back it up at all.

Otherwise get another NAS, installed it at family/friend's house, and set up a VPN between the two: then use rsync/zfs-send/whatever.

Cloud archival tier storage is much cheaper than that now.

Glacier vaults in S3 are quite affordable these days.

LTO5. The cost of my LTO5 system is the cost of downloading all of my data once from a remote cloud provider. It's a nobrainer

For tapes, LTO is really the only game in town, every other tape format is dead. You can get LTO-4 tape drives for dirt cheap because companies have been upgrading them. Yeah, they'll be used, but those drives are meant for heavy duty, and you can just pick up some spares. I found that IBM Fibre-Channel drives are available aplenty, cheap, and they usually come with a front bezel for installation into either two 5.25" slots or something like a Dell PowerVault 114X. (Unlike Library Drives that usually come naked and in non-standard form factors). A FibreChannel host adapter, some cables and transceivers, for probably less than $20 combined, and you're good to go. LTO-4 tapes hold 800 GB and are readily available new for affordable prices as well.

I did upgrade to an LTO-5 drive last year so so, after finding a new-in-box from a liquidation sale for something like $450. The nice thing about LTO is that it's 2 Generations R/W and 3 Generations Read - so the LTO-5 drive will Read/Write LTO-5 (1.5 TB) and LTO-4 tapes, and read LTO-3 tapes. I think with one of the new standards (LTO-8?) it's a bit more muddy, so check compatibility.

I think that LTO-4 and LTO-5 is the sweet spot for hobbyists: You still need to spend some money on a drive or two and buy brand new tapes, but it's reasonably affordable. That said, for a business, I'd just bite the bullet and buy a new drive. Dell sells an external SAS LTO-7 drive brand new for $3700 list price, but I think there might be cheaper options. Together with some tapes and a SAS Controller, I'd say that for $5000 you can get a decent, brand new setup.

I put the tapes in Turtle LTO Cases (https://turtlecase.com/products/lto-20-black), and they sit in a closet. It's not climate controlled or anything, but the place is roughly at a similar temperature year round. The tapes aren't THAT sensitive, but I'd definitely not store them in the garage where I might get a 50+ degree temperature difference throughout the year. That said, there are companies that offer off-site storage options with climate controlled environments. I haven't looked into their pricing since I didn't need it, but the nice thing about tapes is that you can just backup to two tapes and send the second tape off-site. LTO has built-in encryption support, so that's an option.

Twice a year or so, I run a restore of the tape and compare it to the SHA256 that I took while backing up the file (I did build myself some rudimentary cataloging system to SHA256 hash every file, then back it up to tape with tar, and make a record of what file with what SHA256 got backed up when on what tape). I've yet to encounter any bit rot/defective tape issues, but YMMV.

I do use Backblaze's B2 service as well for cold-ish storage. Though I only back up truly irreplaceable or inconvenient to recreate data into B2. That way, I have multiple copies of truly important stuff, I have stuff readily available where I am, and I have terabytes of stuff that isn't worth the expense for the cloud since I can re-create it, but nice to have a copy of.

Tape Drives may be overkill for many and external hard drives (plural!) might be a better option for many. What I like about tape drives is that the media isn't "hot". If I have ransomware running wild, connecting an external hard drive puts everything on it at risk (hence the need for multiple drives), whereas with a tape, it would have to specifically try to rewind the tape and start overwriting, and I would notice it. But YMMV, I never had a ransomware problem myself, but I do have stuff I really don't want to lose, so multiple backups of it in multiple ways (Daily .tar archive on a hard drive, backed up to tape, and backed up to the cloud) should hopefully give defense in depth and the ability to at least recover some older state.