Caveat emptor: I've run ZFS on a two-digit number of machines for >10 years. But no corporate deployment.
To me, a big benefit of ZFS' rampant layering violation is that it shrinks the surface of weird interactions and edge cases to learn about. You get to know the FS, play with it, and then off you go. Most other solutions require a lot of thought and planning. Perhaps there is a better way to tune the storage setup to exactly what you want, but with ZFS you aren't seduced into getting creative with the architecture.
So I'm in favor of native ZFS encryption due to its usability.
That said, it is not as mature as I'd like, and there is at least one very big issue to know about: I recently sent an encrypted FS with zfs send/receive, and the received copy ended up being unreadable. I'll see if I can find the bug again. So that means you should test your setup precisely.
The technical detail from that thread: "This happens because when sending raw encrypted datasets the userspace accounting is present
when it's not expected to be. This leads to the subsequent mount failure due a checksum error when verifying the local mac.
I tried unsuccessfully to tackle this in #11300.
See also: #10523, #11221, #11294.
Edit: If you have critical data lost due to this case I could help you recover them."
I still need to add a couple of new bugs, but almost all of them are dupes or related to one of two flaws - a narrow race in the ARC (...probably), where you can sometimes NULL dereference for fun and sadness, and zfs change-key and send -i resulting in the receiver incorrectly resetting the metadata for which key an encryptionroot is encrypted by.
> So I'm in favor of native ZFS encryption due to its usability.
Another reason to favour it is because if you do ZFS Send/Recieve to do incremental backups (like with rsync.net), you can backup your encrypted datasets to an off-site location without shipping the encryption keys.
That means you don't have to be afraid of your backups somehow resulting in breaking your confidentiality.
So you can still retain your incremental ZFS based backup-flow, and you don't have to change any tooling to use it. With LUKS you will either have to store your backups unencrypted, or you lose the ability to do incremental ZFS-based backups.
I filed #11294, it's a non-issue for me now. Seemed like an API change stability issue, which happens. I think what a crypto-user might care about more in ZFS vs LUKS is precisely what gets encrypted: LUKS can be the entire disk, whereas ZFS is going to need some metadata to know what is there, which keys unlock it, etc on at least a per-dataset basis. It's not a tradeoff that affects my particular threat model, but it's worth understanding as a crypto user.
As a seasoned storage administrator and ZFS veteran, reading over these comments is mildly infuriating. The amount of misconceptions and flagrant untruths is appalling. Using a beta version of FUSE ZoL one weekend 10 years ago does not make you an expert on storage and filesytems. People have no appreciation for what a wonderful gift ZFS is to the storage community, how well architected it is and how hard behlendorf and the rest of the team have worked.
I haven't run ZFS + LUKS but I'm running ZFS Native Encryption.
There is an outstanding issue  that I've encountered on two separate machines running in this setup. The issue occurs when using ZFS Native Encryption + (NVMe?) SSDs. The issue appears to be snapshots getting corrupted (or maybe failing to create?) occasionally. It happens roughly 1/1000 snapshots in my case. I take a lot of snapshots so this occurs weekly.
I haven't lost any data yet but this issue is annoying because I get spooky alerts from ZFS warning me about "potential data corruption". To clear them I have to manually intervene by deleting the corrupt snapshots, restart the machine, then scrub.
What's most annoying is that this breaks ZFS send/recv until I intervene. send/recv was the whole reason I went with native encryption instead of LUKS in the first place.
Again, no data corruption so far (if you don't count the snapshots that get lost, which I don't, because I have so many of them). Just very annoying and tedious.
As others, I'm running ZFS with native encryption on my daily driver Linux box (Arch).
As others have said, I like the benefits, which are easy send/receive, etc (but mind the bugs some other commenter mentioned).
I will also give a few drawbacks / points in favor of using LUKS.
In my case, Arch usually has up-to-date kernels. These are not always compatible with the latest stable OpenZFS release. Issues go from benign: ultra-high CPU usage so my 980 PRO ssd is slower than spinning rust to the module not compiling at all.
Another drawback is that booting with systemd in initramfs doesn't support zfs native encryption. There's no support for using a TPM . So you have to type in your password every time you boot up the computer. If you want hibernation to work, you need a separate swap, encrypted separately, so that's two passwords to type in.
Systemd recently got support for TPM so can automatically decrypt devices. I have this set up on a separate machine (with ext4) and it's nice.
 I know not everyone likes the idea of automatically decrypting the drive, but that's another debate. If your threat model allows for this, it can be done with LUKS but not with ZFS.
Yes it's been resolved. It was around the time kernel 5.16 came out, but it took a while for the fix to land in stable.
I actually followed this closely, since that's when I had got a new Laptop with a new SSD and when I tried to benchmark it, I was going crazy trying to find the culprit. Because an older PC with a slower ssd (and older kernel at the time of benchmarking) was something like 25 times faster.
I've been using ZFS native encryption since it was beta (~2017 I think), and it's nothing short of awesome. It's amazingly convenient and easy to use, and it integrates wonderfully with the rest of the filesystem - in comparison, Btrfs on top of LUKS feels very clunky and cumbersome.
I don't think I'd use it on cold storage though, I like using it on systems I daily drive, such as workstations and laptops, where I feel sacrificing convenience is not worth it, and where I don't want to lose the option to not be able to select what's encrypted and what's not (for instance, I don't want or need for all of my VMs ZVOLs to be encrypted). With ZFS you can pick what gets encrypted, and with what key, without having to fumble around with loopback devices and other strange machinery.
In my experience it is very reliable, I didn't really feel any performance degradation after moving my system on top of encrypted datasets on any of my systems. I also happen to have to use every once in a while a system installed on an external USB-C SSD (with Arch+ZFS), and it works very well without any performance hiccup even with encryption on.
The only time I had to fumble with the dataset was when encryption still wasn't stable and I had to rsync everything to newer format datasets - something around 2018 I guess? Still, it was a pretty straightforward experience. I never had any other issue with encryption after it got moved to stable (3 years ago I think?), daily driving several (personal and work) systems.
On my cold storage, such as drives I backup my stuff onto, I still use LUKS + XFS/ext4/ZFS because it's tested and solid though - I don't really feel that moving disks I often keep offline to ZFS native encryption would help at all. I honestly worry more about them being misplaced or lost, than convenience.
ZFS is still a little on the immature side; there are sufficient recently-fixed bugs involving data-loss that it is hard to be confident that there aren't others yet to be discovered.
That being said, I do use it on a laptop where losing the data wouldn't be a big deal, but having it easily compromised would be. It was just so much easier to transition to encrypted datasets; the whole thing was done with maybe 10 minutes of downtime.
LUKS has its own use case. You can create portable containers with it, which is nice because not everyone wants their OS drive encrypted, sometimes it’s better to be able to move a container from usb stick to drive to whatever.
Shameless plug but i wrote a nice cli script to manage luks containers.
Zfs has a per dataset data encryption key. When you destroy a filesystem or volume, the key is also destroyed. This makes recovery of data in deleted datasets exceedingly hard even with raw disk access.
LUKS and other FDE schemes tend not to have the ability to have fine grained encryption keys. If you have raw access to the LUKS device on which your filesystem resides, you can recover plaintext of blocks that have not been overwritten or erased via TRIM or similar.
zfs on a self encrypting drive seems like a good combination. Even if the Opal implementation is garbage (see CVEs from a couple years back) the exposure is minimal because zfs is encrypting data. My measurements of performance impact IO with and without encryption suggest there is no performance nor increased cpu consumption with self-encrypting drives.
Suppose that I have encrypted ZFS datasets. Naturally, sometimes the process of working with datasets is interrupted (due to, eg, power loss and so on). Is it known how much plaintext could potentially leak to disk outside encrypted ZFS datasets, possibly accessible by forensic analysis?
That would violate the confidentiality guarantee in the event of an accident.
Related to this, if I recall correctly, if the encryption key is changed, ZFS may not destroy the old wrapped key and may keep it stored on disk outside the dataset. This is apparently due to the copy-on-write system.
> Is it known how much plaintext could potentially leak to disk outside encrypted ZFS datasets, possibly accessible by forensic analysis?
When a write to an encrypted dataset happens, no plaintext data is written to disk. Your data is every bit as secure from disclosure as if the write completed normally.
Think of ZFS encryption like you would TLS encryption. They both hide your data, but leak metadata. With ZFS encryption, someone with access to the disk can see what you name your datasets (don't name your dataset "people-i-killed" or name your snapshot @before-starting-the-fire), the order that each block was written (via birth txg in block pointer), how much data was written to which datasets, when the last write happened (via ub_timestamp in uberblock copies in vdev labels), etc. Likewise, with TLS someone with access to the network can tell who the endpoints of the connection are, when each party sent data, and how much data was sent.
If you have full disk encryption under encrypted ZFS, you get the goodness of encrypted ZFS plus after your drive has been powered off, no one can determine anything about the content of the drive (except maybe that which is leaked via SMART and similar data). This is kinda like having a VPN and using TLS. Someone with access to the network under the VPN can still see that there's activity but has a much harder time figuring out information about the data and metadata with the VPN.
ZFS encryption on a disk can perform better than LUKS since LUKS has integrity enabled by default and TRIM/discard can't be used. If you encrypt the full nvme this can lead to premature drive failures and loss in performance.
Not sure where you are getting that. Lots of former Sun people now work on the OpenSource Solaris. Matt Ahrens was one was the Nr.2 guy who did most of the implementation and he is still heavily involved in OpenZFS.
OpenZFS has a very healthy and active community working on it.
And the code has been much improved compared to originally open sourced version by Sun. Technology has moved on since then and so has ZFS.
Sun had certain values as a company it enforced through its corporate culture which made ZFS excellent.
My perception is that the ZoL based fork had a lot more instability and bugs compared to the parallel development of ZFS by in the Open Source Solaris world. That OpenZFS is based on the more unreliable code base should be evidence that the values which made ZFS excellent under Suns steermanship are insufficiently maintained and that features integrated into ZFS after the fork who did come from the ZoL side should not be used if one wants the reliability of ZFS.
The fact that not all metadata is encrypted shows they are doing things wrong from a principled standpoint. Sun didn't halfbake important stuff like that. The ZoL community will in the long run make the reliability of OpenZFS converge against the reliability of BTRFS because that is the level of reliability open source allows for. Initially higher stability be dammed.
Many of the open source zfs developers are the same ones that developed zfs at Sun and then Oracle. I worked on the Solaris team from 2010 until 2017 and have contributed to open source zfs implantations in illumos and openzfs. Your criticisms seem ill-informed based on my actual experience in my contributions to these multiple zfs code bases. I actually found a fair number of improvements in the testing in ZoL.
With Solaris zfs, you saw features after they were exercised by up to a few hundred engineers for months to years. Now you are getting bits fresh from the factory. Like when software is developed behind closed doors, bugs happen and they are progressively squashed leading to a more stable product.
Everything outside of encrypted datasets’ data is not encrypted. This includes the things you mention as well as vdev labels, block pointers, etc. The clear text bits can tell you a fair amount about the relative age of written data and how much data is present. It will also identify which machine last imported the pool.
And when something breaks (maybe because one of those numerous bugs) there are no proper tools for repair. Your ZRAID and ZLOG probably won't matter. The to-go tool for removing corrupt superblocks is a 10 year old hacky python script, it's ridiculous.
I have seen folks get in over their heads though. Like -- "OMG I've somehow broken boot for ZFS on root and I don't know how to fix!!" or "I'm using a custom kernel and OMG things don't work!!"
Canonical has even been complicit. When you download an HWE kernel, you don't also download a newer/matching versions of the tools. This should pretty clearly be considered a bug, and fixed but Canonical perhaps doesn't have the bandwidth?
I just can't lay any of these problems at the feet of ZFS though. Right now I trust ZFS a hell of a lot more than I trust Linux to Do The Right Thing. When ZFS seems broken (though I have little experience with native encryption so this may not apply to it), I am far more likely to think the issue is Linux, my distribution, or me, than any problem with ZFS.
The last time I needed to use fsck to repair a filesystem was less than six months ago. After running it I found that my entire directory hierarchy was garbage and lost+found had at least hundreds of files named by their inode numbers. It would have taken me forever to piece this back together if it was a filesystem that I cared about. This is consistent with my experience with fsck in decades past.
In 15 years of using zfs, I’ve never lost a pool when it was in a mirror or raidz layout.
If you stream your ZFS data from one host to another then the encrypted filesystem remains encrypted at the destination.
If your encryption is done outside of ZFS then the data on the destination is in plaintext.
This might seem obvious to those experienced but to a newcomer it caught me off guard and the interaction between recursive send/receive and encryption was more complicated than I had expected.
I should add that I haven’t looked into this for a while and could be completely wrong but it’s worth researching, at least, because my spideysense is definitely tingling to the tune of something about this was odd and worth knowing about.
In addition to your points, if you intend to back up your ZFS datasets, it's easier with ZFS encryption to use ZFS commandline tools (zfs send) to ship the whole encrypted dataset over the wire. With LUKS you're shipping the unencrypted dataset instead.