The ZFS-equivalent on Windows is meant to be ReFS, which is roughly equivalent to ZFS when layered on top of Storage Spaces.
I'm really keen to give ReFS+SS a go, but I keep hearing horror stories of abysmal parity-spaces performance, which can only be explained by poor coding and testing.
Even if ZFS has better performance, it's never going to have full support, and it wasn't written for the NT kernel IO model, so I'm dubious about it ever working well...
I once tried ReFS, back on Windows 8.1 IIRC. Everything was going fine until suddenly, in the middle of the day, Windows decided that the ReFS volume was corrupt. And then you discover there’s virtually no rescue tools. It nearly gave me a heart attack. After rebooting three times it decided everything was okay again.
I quickly moved away my data and never used ReFS since.
This entirely misses the fact that zfs (open zfs/zfsonlinux) is open and cross platform.
Which is almost more interesting than all the good stuff that comes with zfs (volume management, disk management, software raid-like functionality, encryption, snapshots, send/receive).
Its the first time we truly have a half-modern filesystem with safe write support and sane performance across many different os's. Even if just made dual booting freebsd and Linux this would be a fmgreat win imnho.
Unfortunately, FreeBSD uses different ZFS then Linux. Yes, it's crazy, but currently I think it is not guaranteed to be able to mount ZFS between these systems. Fortunately, FreeBSD is supposed to base their ZFS on Linux version in future.
ZFS is mountable between Linux and FreeBSD so long as optional features are configured in an appropriate way - this is a benefit of OpenZFS "version 5000" scheme.
IIRC, one of the issues involves the layout of extended attributes. FreeBSD keeps to Solaris layout (which is essentially the same in concept to NTFS' one), while Linux for speeding up certain stuff (and because Linux can't use native xattrs to full extent anyway, due to using Irix API which is limited to PAGESIZE xattrs) has an optional feature to pack xattrs more densely.
I used zfs on Linux with encrypted / for a while on Debian. It works really well, but unfortunately when I upgraded from Jessie to Buster, it did not recognize how I set it up and broke the system. I ended up having to reinstall.
BTW, did you do a traditional in-place upgrade via apt? I've once or twice managed to break boot between Debian releases.. But not since lilo/early grub days.
Sounds like you probably ran into zfs not being in Debian proper - and buster unlocking dmcrypt, but not being able to mount zfs? Should normally be salvageable via booting previous Jessie kernel and initial ram/root disk (initrd).
At any rate, I gather you've moved on - just sound like an unlucky break - even with root on unsupported zfs.
Yeah I did an in-place upgrade via apt. I for some reason thought zfs was in debian proper (contrib vs. main). Like I said in another post, I may try doing that again.
But then again, the main reason I haven't is because all of my stuff is backed up via Nextcloud, so reinstalling Debian is pretty quick/painless for me.
Related: I moved to a new laptop at work with a zfs send over ssh of /home/myuser - and a mount+manual copy of what I wanted. Very straight forward - but not that different from an rsync to /home/myuser/olfstuff.
Zfs encryption was still a bit new, so both boxes uses dmcrypt on full disk plus efi plus /boot (technically redundant, but it works).
Looking forward to hopefully root on encrypted zfs in next Ubuntu lts.
To be honest I didn't feel like tinkering with it, I just reinstalled. Right now it is on ext4.
My issue was that for some reason Linux no longer could find the encrypted partition to decrypt, so I couldn't get it up to a point where I could even work with it.
Ah that's a shame. If you'd had a bootable USB stick or some pre-boot environment to use, you probably could have rolled back from a snapshot. At least, that's my understanding of the theory.
Your experience has definitely not discouraged me from wanting to ZFS my root partitions though. If only to provide the potential / theoretical ability to roll back a snapshot if necessary!
To be honest, it was laziness on my part, I could have probably figured it out and fixed it. The snapshot ability was incredibly nice, don't get me wrong. I really liked it on my system. I may switch back in the near future.
I can post my "script" when I get home if you want to try it. I use script very liberally, it was my notes of the commands I used in order in order to get ZFS/dmcrypt working.
I've used (conceptually equivalent) btrfs snapshots to let me install arch and gentoo, always easily rolling back to my own heavily customized Ubuntu system. I've never tried it, but given a copy of the snapshot tools, you should even be able to rollback from formatting your root partition.
(ed: there are some potential benefits to zfs encryption - it's as cross platform as zfs (encrypt an USB drive, use on Linux, os x, windows, freebsd).
Zfs send/receive - receiver can take your encrypted backup - key doesn't have to travel.
Mount drives across dual boot - share encrypted /home between dual boot windows and Linux?
Not sure - but I think volume encryption is supported too. Might simplify encrypted swap (if you already "pay" for zfs). Note: there are apparently some issues with swap on zfs (volumes at least, possibly files too)
If you don't need suspend/resume - suppose you could share one encrypted swap volume across multiboot environments..)
)
Like I said elsewhere, I could have probably fixed it, but I was lazy. I also wanted to try out Wayland, but I ended up going back to Xfce. Maybe when Xfce moves to Wayland I will try again.
Isn't encryption supposed to be a separate layer from the filesystem? Or is the merger specific to Windows? Or, is this ZFS' attitude toward features, like with the volume manager?
ZFS "native encryption" is an upstream feature. The argument for doing it in the filesystem rather than as a separate layer is that it allows you to have far more flexibility than classic FDE (encrypted send/recv, you can use different keys for each filesystem, the encryption applies to all data written by ZFS so you don't need to worry about forgetting to set up LUKS on a new drive added to your pool, etc.)
One more addition to other replies: ZFS needs access to disks for self-healing (bit-rot detection and repair). It means that in the past if we wanted encryption, same date needed to be encrypted twice (one for every mirror).
Now with encryption in ZFS, date is encrypted before creating redundancy. We can even do a scrub without encryption key.
In addition to what @cyphar mentions - zfs is "storage manager". Disk, volume and filesystem. Some of the power and data guarantees relate to this design decision.
Encrypted /home/username filesystem with user-owned keys, is one neat trick,for example.
Oracle "closed" (forked) zfs "back" and added encryption. But encryption is (now) in openzfs (who's canonical upstream has gravitated to "zfsonlinux" which is just the open project that ended up having the major mind-share).
Instant backups (snapshots) alone are worth the investment.
Take a snapshot before you update your kernel, and poof super-simple recovery.
Working on a project and need to access it remotely? Take a snapshot and then pull it to wherever you are, your dev environment anywhere. (Different use-case from GIT.)
Snapshot /home/ and worry less about computer migrations.
ZFS is _THE_ power-user tool of the decade hands down.
Disclaimer: I run FreeBSD on my workstation, laptop, and servers. I've never tried to "mix & match" FreeBSD & Linux ZFS implementations.
I run Linux with ZFS on my home NAS and I will echo your sentiments. Accidentally deleted some photos? They're all in Photos/.zfs/2019-10-14/. I can't wait to run ZFS on my desktop too.
I use it on both servers and notebooks. I did not notice increased power usage, but if it matters to you, you should check it.
I use it mainly for non-root volumes because of bad experience in the past (dkms deciding to not build zfs module on first try after update, kernel incompatibility, grub incompatibility, etc.). It's better now, but unless you are experienced I would suggest to not use it for root.
I am using it on my mini server for around 10 years (freebsd 3x3tb as zraid + 10tb disk for casual things) and now on laptop as a root for fedora and I wouldnt part from it. It is the only fs with snapshot capabilities that i fully trust. It didnt crash even once. It is battle proven and I never understood why distributions are not including it as a default or at least as a part of basic installation. Although I would avoid encryption for a few years. Just for any case.
One thing if you start playing with it (I am surprised how little users know about this feature): you can access the snapshot content using hidden "directory" /.zfs/snapshot/snapshotname ;) Just cd into it.
One word of warning, near end of pool capacity it will start to fragment (there is no defragmenter for it, but if they have made implementing encryption, defragmentation shouldnt be far away) and performance will drop. Dont let it happen, always keep 10-20% free and dont forget about old snapshots size.
I used it under BSD for ages, and started using it in Linux many years ago, first as FUSE, then ZOL. It's rock solid.
The only thing is that it "uses" all your RAM, but gives it back as needed. Also, if your volume gets over 90% full, it starts becoming VERY inefficient.
I wouldn't put zfs anywhere except a server or non root volume, the resource usage is high and if I had to guess it would have an impact on battery life.
Unleas you're into either file systems or installers as a hobby, let others be the early adopters.
>ZFS is already quite old filesystem as has great features (for example: checksumming, snapshots, compression) that should be already standard.
Sure but if you don't know how zfs works do you really want to figure it out in the middle of trouble shooting installation problems, or post install root volume problems?
>Isn't it similar to using mdraid, lvm and dm-crypt without experience?
Those are all pretty different, but yeah basically we agree on this. I'm not sure about battery life, if I had to give a more nuanced answer I would say it probably depends on use case and you should measure it if you want to be sure.
Since I started seriously using it for my home server, I decided that I never want to use any other file system on any of my machines, it's that good. I use it on my desktop machine and will never look back. Snapshotting can be beneficial for any kind of machine for example.
I think it's neat that having the possibility of a ZFS root on Ubuntu exists, but I just don't see myself using it. Will anyone, and if so for what purpose?
I'm not in the home NAS building sphere, but wouldn't you rather have your root be a SquashFS image on an SD card or similar, with the storage section being ZFS?
As for my laptop, I guess the snapshots are nice. I've never had a situation with my laptop where I wished I had snapshots, and even if something did break I only care about the files I have, which I back up. Then the snapshots are an extra safety net I suppose.
The first is generic: I work only on filesystems with data checksumming; the current ones are BTRFS and ZFS, and I have a low opinion of the developers/development of the former.
The second is specific: I keep my disks encrypted; since v0.8, ZFS has encryption integrated, which allows me to have disks encrypted and mirrored without additional layers (BTRFS requires LVM for this setup).
In the past I've also used snapshots, by the way, which are really functional.
ZFS on root enables you to have boot environments, which is a functionality originally found on Solaris. Not that they're in scope for this initial implementation in Ubuntu, but they could be built on top.
Furthermore, ZFS protects you from bitrot via multiple copies or RAIDZ or at least lets you know bitrot occurred (scrub). It lets you resize the dataset (I tend to undersize partitions such as /boot). I could probably think of other reasons as well, but this would be enough for me.
Me and few of my friends have been using ZFS on laptops for few years now (since 2013 on a laptop for me, since 2012 on a workstation).
There's no real increase in power usage except for one thing, which is configurable - the transaction group commit frequency (I leave it default, a friend of mine sets it to very powersaving level coupled with running "nosync").
A quick description is that TXG write period says how often ZFS writes data that was written without syncing to disk, and commits data that was written synchronously. Every "transaction group" is essentially a snapshot, and by default it's done every 5 seconds, or every ~10? MBs written, whichever is first. Data that is written synchronously between that is written into "ZFS Intent Log", and all data written is kept in memory without sending to disk (except for sync writes).
If you disable sync writes (datasets mounted sync=none), ZFS will only wake up drives for writing with the interval set in this parameter, or when you start writing significant amounts of data (where "significant" is also configurable).
If you're willing to lose X seconds of writes, you can configure sync=none and transaction writes to happen every X seconds. My friend does that, myself I prefer my data safer :)
That's the story on power usage with ZFS. I'd recommend maxing out memory where possible, but I recommend that even without ZFS.
Then there are benefits, like recovery from snapshots (more than once I recovered a file I accidentally overwrote because I missed a '>' in '>>'), easy backup/migration with zfs send/recv. I also found it pretty easy to install multiple linux distributions using the same root pool, by giving them different root datasets and putting home directory and similar on separate dataset (on the same pool).
This allowed pretty nice migration between distributions, so long as I kept UID/GID of my user the same between them.
> but wouldn't you rather have your root be a SquashFS image on an SD card or similar, with the storage section being ZFS?
Read-only root filesystems are one of those things that everybody ought to do, yet are usually impractical. Unfortunately, most software doesn't really support the Linux Filesystem Hierarchy Standard, so you have software that modifies itself under /usr (supposed to be read-only), ever-growing logs and caches not stored under /var/log and /var/cache so they can't be managed automatically, etc.
You generally find squashfs roots where the distro is specialized and not general-purpose - for a recent example, see Talos.
Fedora Silverblue aspires to implement read-only root for a general-purpose distro. The main cost of their implementation is that you have to reboot after installing or upgrading packages to make the changes effective.
The OSs that insisted upon scattering application files all over the hierarchy only have themselves to blame for this situation. At some point in the past 30 years thought could have been given to the idea that applications are separate from the platform they run on, but UNIX just doesn't think that way. Everything must be part of the same gigantic Goldberg-esq state so everything can catch fire and burn down at once as soon as there's a conflict of any kind.
Anyway, the fix is pretty obvious: put each application in its own namespace, thus its view of "/usr" isn't the same as the immutable OS "/usr" and it is free to write whatever it wants without causing any trouble.
> Everything must be part of the same gigantic Goldberg-esque state so that everything can catch fire and burn down at once as soon as there's a conflict of any kind.
Well, that's exactly the job of competent package managers, to make sure that packages fully stated their dependencies (so that the package manager would forbid you from getting into an illegal state wherein a required dependency was not installed or the wrong version of the dependency was installed), and installed correctly according to the LFHS so that there would be no conflicts.
/opt isn't new. Plenty of people decided that they could throw everything into a folder under /opt, including copies of all dependencies, with no references outside of that folder, and that would be enough. And it might have been enough if it wasn't for sysadmins in charge of uptime tearing their beards out over /opt constantly filling up and causing multi-product outages when it did.
The whole point of the LFHS is that it represented a contract along the boundary between developers (who needed to bundle libraries, save state, have access to a cache, etc.) and operators (who worried about disks and associated costs - why would you ever spend a dime backing up /var/cache? Why would you ever use a fast disk for /etc?). When this contract was followed, systems worked splendidly. When the contract was broken, the people who broke the contract shot themselves in the foot just about the same as anybody writing C code knows how many footguns are in the C language.
Of course, today, with the price of resources having crashed through the floor, and containerization having more or less reached maturity, this is all more or less an academic, historical, moot point. But dynamically linked libraries weren't stupid, they were an artifact of an age where statically linking libraries was horrendously uneconomical and the idea of namespacing anything was enough to get laughed out of the room by people wondering how you would expect to pay for any of it.
> Well, that's exactly the job of competent package managers, to make sure that packages fully stated their dependencies (so that the package manager would forbid you from getting into an illegal state wherein a required dependency was not installed or the wrong version of the dependency was installed), and installed correctly according to the LFHS so that there would be no conflicts.
Package managers are an over-engineered solution to a problem that only really exists if you rely on package managers. Ever had your apt database break on you? It's a joy, let me tell you. Nothing ever gets installed or removed again. Blowing up your system and requiring your OS to be reinstalled is ridiculous. Especially since that was a problem that didn't even used to exist. OSs were once stored on roms.
> /opt isn't new. Plenty of people decided that they could throw everything into a folder under /opt, including copies of all dependencies, with no references outside of that folder, and that would be enough.
Sure, as long as you never move it from opt. UNIX devs' addiction to hard-coding fixed paths is difficult to break.
> And it might have been enough if it wasn't for sysadmins in charge of uptime tearing their beards out over /opt constantly filling up and causing multi-product outages when it did.
Is that really a problem outside of Linux, which famously defined its base-system as "the kernel" and therefore necessitated that any such install would need to bring along all its own dependencies because it couldn't rely on anything existing except the kernel?
> Of course, today, with the price of resources having crashed through the floor, and containerization having more or less reached maturity, this is all more or less an academic, historical, moot point. But dynamically linked libraries weren't stupid, they were an artifact of an age where statically linking libraries was horrendously uneconomical and the idea of namespacing anything was enough to get laughed out of the room by people wondering how you would expect to pay for any of it.
Yet we still live with this bullshit today anyway and the Linux community especially eschews any attempt to move on. This despite some of the same people who created UNIX creating Plan 9, which rejected dynamically linking and embraced namespacing over twenty years ago!
> At some point in the past 30 years thought could have been given to the idea that applications are separate from the platform they run on, but UNIX just doesn't think that way. Everything must be part of the same gigantic Goldberg-esq state so everything can catch fire and burn down at once as soon as there's a conflict of any kind.
this only happens if one isnt aren't managing applications coherently (e.g binary only packages managed haphazardly, like most linux binary packagers)
on systems where there is a clear base+add-ons distinction (e.g. source based ports), these things happen much less often - all packages are built from a coherent source tree, and read only root is often feasable because packages only install reference files into their respective '/usr/share' area and don't muck with the 'real system' in /etc - the latter being an administrative action (e.g. package installation is a separate action from service configuration / execution)
So basically, all software has to be built from source by maintainers who are careful to make sure it doesn't conflict with other software. And people wonder why applications aren't ported to these OSs.
The OS should be a platform. I should be able to build an application and distribute it to users and have it just run on that platform without involving third party middlemen and regardless of other software they have. You know what OSs accomplished this? The original Macintosh, RiscOS, DOS, and (mostly) Windows 3.1. Why is it that we can't accomplish today something we did 30 years ago?
a platform for 'building your system', or a platform for 'distributing and executing binaries'?
the binary only distribution mentality comes from microcomputer culture, which postdates unix/mini culture and is rooted in non-technical users installing purchased 'application' programs from vendors rather than computer users (read: programmers) building their systems to suit their own personal data processing tasks.
you highlight this in your choice of platforms 'getting this right', but really, these are two different mentalities, cultures, and modes of operation.
not saying one is superior to the other, but square peg, round hole.
and besides:
many proprietary programs for unix just ship with static executables.
problem solved.
> a platform for what?
> a platform for 'building your system', or a platform for 'distributing and executing binaries'?
These are not mutually exclusive goals.
> the binary only distribution mentality comes from microcomputer culture, which postdates unix/mini culture and is rooted in non-technical users installing purchased 'application' programs from vendors rather than computer users (read: programmers) building their systems to suit their own personal data processing tasks.
"Non-technical". Elitist bullshit. Woz was non-technical? He was clearly a micro guy. People got work done with micros. They made things with micros. Like your glorious programmer master-race, they configured their environment into something that enhanced their lives. They bought and used software the way you import 10000 dependencies. Excel is still by far the most used programming environment in the world, with entire businesses run on tooling never touched by programmer-master-race hands.
Besides, programmers should want it this way too. Which makes more sense to you? A constantly mutating state where changes in seemingly unrelated portions can wreck havok on the whole, or a collection of immutable components which can be swapped in and out at will? A system so complicated it needs sophisticated management software and a database of conflicts and dependencies to keep it all running, or one so simple that 'copy' and 'install', and 'delete' and 'uninstall' are synonymous?
If the UNIX model is so damn great, why the fuck does Docker exist!? If development is so damn great on UNIX, why are people building and distributing dev environments in containers!?
> you highlight this in your choice of platforms 'getting this right', but really, these are two different mentalities, cultures, and modes of operation.
> not saying one is superior to the other, but square peg, round hole.
I disagree. I think one is clearly objectively superior to the other.
> many proprietary programs for unix just ship with static executables. problem solved.
Ha! Yeah. Problem solved. Even for proprietary software with a native Linux version I've almost always had better luck running the Windows version under WINE.
I run NixOS as a daily driver, and while I'm aware that the install media is a squashfs mount and that it's feasible to create server images where /nix/store is a squashfs mount as well. But in day-to-day use on a development machine, running nixos-rebuild switch makes a read-only root impractical, so far as I know.
Will you detect data corruption before your snapshots rotate? ZFS controls saved data. Also, it saves more more space then rsync as it can store just the difference (rsync will copy whole file if changed, which makes things like snapshots of virtual machine very impractical).
Also, being able to snapshot the entire system at regular intervals, show diffs between snapshots, and then perhaps do security analysis on the results might be handy.
I use ZFS on root on Ubuntu for its flexibility and relative ease of use. It’s easy for example to set different compression ratios on various paths by creating arbitrary volumes. Plus the checksum protection is nice. You can use snapshots for backups too.
My biggest complaint with ZFS is the lack of flexibility. If you setup two disks in a mirrored configuration, that's the end of it - those disks will be mirrored forever. If you get another drive and want to expand storage while still having redundancy, i.e. RAID5 type setup, you can't. You can increase storage with a pair of drives, by mirroring them first and then adding them to the zpool, but this is not an option configuration from a safety standpoint.
I loved the flexibility of BTRFS in this regard - you could change the duplication level at any point. BTRFS seemed to abstract the storage away beautifully - just give us your drives, any combination of sizes, tell us how many copies of your data you want to keep and we'll handle the rest. It wasn't perfect, but I loved this approach, and I feel like it could go even further, eg. calculate risk of failure probabilities and then just tell it how low you want your risk. If it can't do it with your current drive capacities it could tell you what it needs to accomplish it, and then rebalance the data once you add the drives. Got an SSD? Throw it in the pool and let it optimize for cacheing. I know bcachefs has that particular aspect covered, but there's a lot of other basic features that still need to be implemented.
for zfsonlinux As far as I understand you can currently add and remove striped/mirror devices, but you cannot remove raid-z vdevs (ref: https://github.com/zfsonlinux/zfs/issues/9129) - so there is definitely work left to be done in this area.
as an aside mdadm has surprisingly fantastic features in this area and can reshape pretty much any array type to any other. i was impressed by that.
Strangely, rather old ZFS keeps popping up in the news, while I never hear of Btrfs these days. It had some hype five or so years ago―what became of it?
That's fine. However file system that I can even configure or use in a way that eats my data, I will simply not consider using. And I will not consider using OpenSuse because of that.
When it was eating my data it was also supposed to be stable. But turns out it wasn't.
Yeah I tried restoring, non of the common tools or tricks I found online at that time helped me. If this particlar video would have made the difference then that's still a shitty situation.
If you want to use btrfs go ahead, I will not do so again.
Used. Only the orignal Jolla phone introduced at the end of 2013 used btrfs. There have been many many problems with btrfs on the device, mainly due to metadata vs data block allocation issues & resulting metadata space exhaustion. The balance ooeration being basically unbounded in time and fregvently crashing outright did not help.
Since then all other devices made or officially supported by Jolla are using EXT4 on LVM, including this one Xperia X running Sailfish OS I'm typing this on.
Huh, I could have sword the Xperia X port also used BTRFS, but I guess I never checked.
(And yeah, those issues were how I found out: unable to create files while having "4 gb of free space" - not helped by the fact that it didn't run the balancer automatically.)
It would be great to know if I could boot with mirrored and encrypted disks.
What happens if one of the disks fails? Is there an easy way to boot from the other disk (I.e. do they replicate the MSDOS/UEFI) boot partition?)
"By the numbers: ZFS Performance Results from Six Operating Systems and Their Derivatives" https://www.youtube.com/watch?v=HrUvjocWopI
Slides: https://callfortesting.org/log/vBSDcon-2019-Dexter-BytheNumb...
And was surprised that zfs for windows (and hyper-v core) appear to be alive and comming along.
Anyone using zfs on windows? Are they also going off of zfsonlinux - and if so how's support for encryption comming?
Finally having shared, read write, encrypted home that works on both Linux and windows safely would be great!
I'm really keen to give ReFS+SS a go, but I keep hearing horror stories of abysmal parity-spaces performance, which can only be explained by poor coding and testing.
Even if ZFS has better performance, it's never going to have full support, and it wasn't written for the NT kernel IO model, so I'm dubious about it ever working well...
Sincerely, someone who had to port kernel level code from Linux to NT.
I quickly moved away my data and never used ReFS since.
Which is almost more interesting than all the good stuff that comes with zfs (volume management, disk management, software raid-like functionality, encryption, snapshots, send/receive).
Its the first time we truly have a half-modern filesystem with safe write support and sane performance across many different os's. Even if just made dual booting freebsd and Linux this would be a fmgreat win imnho.
If the choice is between zfs and fat32...
IIRC, one of the issues involves the layout of extended attributes. FreeBSD keeps to Solaris layout (which is essentially the same in concept to NTFS' one), while Linux for speeding up certain stuff (and because Linux can't use native xattrs to full extent anyway, due to using Irix API which is limited to PAGESIZE xattrs) has an optional feature to pack xattrs more densely.
Sounds like you probably ran into zfs not being in Debian proper - and buster unlocking dmcrypt, but not being able to mount zfs? Should normally be salvageable via booting previous Jessie kernel and initial ram/root disk (initrd).
At any rate, I gather you've moved on - just sound like an unlucky break - even with root on unsupported zfs.
But then again, the main reason I haven't is because all of my stuff is backed up via Nextcloud, so reinstalling Debian is pretty quick/painless for me.
Zfs encryption was still a bit new, so both boxes uses dmcrypt on full disk plus efi plus /boot (technically redundant, but it works).
Looking forward to hopefully root on encrypted zfs in next Ubuntu lts.
Curious if snapshots can protect you even as far as OS upgrades gone wrong. I'd assume they can, if all filesystems are ZFS + snapshotted.
My issue was that for some reason Linux no longer could find the encrypted partition to decrypt, so I couldn't get it up to a point where I could even work with it.
Your experience has definitely not discouraged me from wanting to ZFS my root partitions though. If only to provide the potential / theoretical ability to roll back a snapshot if necessary!
I can post my "script" when I get home if you want to try it. I use script very liberally, it was my notes of the commands I used in order in order to get ZFS/dmcrypt working.
(open) Zfs now has native encryption:
https://github.com/zfsonlinux/zfs/releases/tag/zfs-0.8.0
(ed: there are some potential benefits to zfs encryption - it's as cross platform as zfs (encrypt an USB drive, use on Linux, os x, windows, freebsd).
Zfs send/receive - receiver can take your encrypted backup - key doesn't have to travel.
Mount drives across dual boot - share encrypted /home between dual boot windows and Linux?
Not sure - but I think volume encryption is supported too. Might simplify encrypted swap (if you already "pay" for zfs). Note: there are apparently some issues with swap on zfs (volumes at least, possibly files too)
If you don't need suspend/resume - suppose you could share one encrypted swap volume across multiboot environments..) )
Why?
One more addition to other replies: ZFS needs access to disks for self-healing (bit-rot detection and repair). It means that in the past if we wanted encryption, same date needed to be encrypted twice (one for every mirror).
Now with encryption in ZFS, date is encrypted before creating redundancy. We can even do a scrub without encryption key.
Encrypted /home/username filesystem with user-owned keys, is one neat trick,for example.
The ZFS specific encryption IIRC is owned by Oracle, and they do not open source it.
https://github.com/zfsonlinux/zfs/releases/tag/zfs-0.8.0
Instant backups (snapshots) alone are worth the investment. Take a snapshot before you update your kernel, and poof super-simple recovery. Working on a project and need to access it remotely? Take a snapshot and then pull it to wherever you are, your dev environment anywhere. (Different use-case from GIT.)
Snapshot /home/ and worry less about computer migrations. ZFS is _THE_ power-user tool of the decade hands down.
Disclaimer: I run FreeBSD on my workstation, laptop, and servers. I've never tried to "mix & match" FreeBSD & Linux ZFS implementations.
I use it mainly for non-root volumes because of bad experience in the past (dkms deciding to not build zfs module on first try after update, kernel incompatibility, grub incompatibility, etc.). It's better now, but unless you are experienced I would suggest to not use it for root.
Main reasons I use it:
* checksums - I had undetected data corruption
* snapshots - great addition to backups
* compression
One thing if you start playing with it (I am surprised how little users know about this feature): you can access the snapshot content using hidden "directory" /.zfs/snapshot/snapshotname ;) Just cd into it.
One word of warning, near end of pool capacity it will start to fragment (there is no defragmenter for it, but if they have made implementing encryption, defragmentation shouldnt be far away) and performance will drop. Dont let it happen, always keep 10-20% free and dont forget about old snapshots size.
The only thing is that it "uses" all your RAM, but gives it back as needed. Also, if your volume gets over 90% full, it starts becoming VERY inefficient.
I’m running FreeNas at home with 6x4TB WD Red drives. All my computers back up to FreeNas. My Plex server reads/writes all its media from it as well.
I’m going to switch FreeNas to a Linux server with ZFS at some point.
Checksums, compression, snapshots (so easy and fast incremental backups using send/receive).
ZFS is much more then RAID. Also, even with only one drive we can increase our chances a bit using copies=2.
Data validation, snapshots, compression, zfs send (for backups)?
Unleas you're into either file systems or installers as a hobby, let others be the early adopters.
I guess it saves battery due to ARC (adaptive replacement cache).
> Unleas you're into either file systems or installers as a hobby, let others be the early adopters.
ZFS is already quite old filesystem as has great features (for example: checksumming, snapshots, compression) that should be already standard.
Sure but if you don't know how zfs works do you really want to figure it out in the middle of trouble shooting installation problems, or post install root volume problems?
Isn't it similar to using mdraid, lvm and dm-crypt without experience?
> or post install root volume problems?
In my other comments I suggested to not use ZFS for rootfs unless someone is already experienced.
Those are all pretty different, but yeah basically we agree on this. I'm not sure about battery life, if I had to give a more nuanced answer I would say it probably depends on use case and you should measure it if you want to be sure.
I'm not in the home NAS building sphere, but wouldn't you rather have your root be a SquashFS image on an SD card or similar, with the storage section being ZFS?
As for my laptop, I guess the snapshots are nice. I've never had a situation with my laptop where I wished I had snapshots, and even if something did break I only care about the files I have, which I back up. Then the snapshots are an extra safety net I suppose.
The first is generic: I work only on filesystems with data checksumming; the current ones are BTRFS and ZFS, and I have a low opinion of the developers/development of the former.
The second is specific: I keep my disks encrypted; since v0.8, ZFS has encryption integrated, which allows me to have disks encrypted and mirrored without additional layers (BTRFS requires LVM for this setup).
In the past I've also used snapshots, by the way, which are really functional.
Scrub is not even required for that: all data is checked when read, it's only for checking unused data.
> It lets you resize the dataset
Considering that you are talking about quota on directories, I'm not sure it is really advantage of ZFS. It's basic functionality.
There's no real increase in power usage except for one thing, which is configurable - the transaction group commit frequency (I leave it default, a friend of mine sets it to very powersaving level coupled with running "nosync").
A quick description is that TXG write period says how often ZFS writes data that was written without syncing to disk, and commits data that was written synchronously. Every "transaction group" is essentially a snapshot, and by default it's done every 5 seconds, or every ~10? MBs written, whichever is first. Data that is written synchronously between that is written into "ZFS Intent Log", and all data written is kept in memory without sending to disk (except for sync writes).
If you disable sync writes (datasets mounted sync=none), ZFS will only wake up drives for writing with the interval set in this parameter, or when you start writing significant amounts of data (where "significant" is also configurable).
If you're willing to lose X seconds of writes, you can configure sync=none and transaction writes to happen every X seconds. My friend does that, myself I prefer my data safer :)
That's the story on power usage with ZFS. I'd recommend maxing out memory where possible, but I recommend that even without ZFS.
Then there are benefits, like recovery from snapshots (more than once I recovered a file I accidentally overwrote because I missed a '>' in '>>'), easy backup/migration with zfs send/recv. I also found it pretty easy to install multiple linux distributions using the same root pool, by giving them different root datasets and putting home directory and similar on separate dataset (on the same pool).
This allowed pretty nice migration between distributions, so long as I kept UID/GID of my user the same between them.
Read-only root filesystems are one of those things that everybody ought to do, yet are usually impractical. Unfortunately, most software doesn't really support the Linux Filesystem Hierarchy Standard, so you have software that modifies itself under /usr (supposed to be read-only), ever-growing logs and caches not stored under /var/log and /var/cache so they can't be managed automatically, etc.
You generally find squashfs roots where the distro is specialized and not general-purpose - for a recent example, see Talos.
Anyway, the fix is pretty obvious: put each application in its own namespace, thus its view of "/usr" isn't the same as the immutable OS "/usr" and it is free to write whatever it wants without causing any trouble.
Well, that's exactly the job of competent package managers, to make sure that packages fully stated their dependencies (so that the package manager would forbid you from getting into an illegal state wherein a required dependency was not installed or the wrong version of the dependency was installed), and installed correctly according to the LFHS so that there would be no conflicts.
/opt isn't new. Plenty of people decided that they could throw everything into a folder under /opt, including copies of all dependencies, with no references outside of that folder, and that would be enough. And it might have been enough if it wasn't for sysadmins in charge of uptime tearing their beards out over /opt constantly filling up and causing multi-product outages when it did.
The whole point of the LFHS is that it represented a contract along the boundary between developers (who needed to bundle libraries, save state, have access to a cache, etc.) and operators (who worried about disks and associated costs - why would you ever spend a dime backing up /var/cache? Why would you ever use a fast disk for /etc?). When this contract was followed, systems worked splendidly. When the contract was broken, the people who broke the contract shot themselves in the foot just about the same as anybody writing C code knows how many footguns are in the C language.
Of course, today, with the price of resources having crashed through the floor, and containerization having more or less reached maturity, this is all more or less an academic, historical, moot point. But dynamically linked libraries weren't stupid, they were an artifact of an age where statically linking libraries was horrendously uneconomical and the idea of namespacing anything was enough to get laughed out of the room by people wondering how you would expect to pay for any of it.
Package managers are an over-engineered solution to a problem that only really exists if you rely on package managers. Ever had your apt database break on you? It's a joy, let me tell you. Nothing ever gets installed or removed again. Blowing up your system and requiring your OS to be reinstalled is ridiculous. Especially since that was a problem that didn't even used to exist. OSs were once stored on roms.
> /opt isn't new. Plenty of people decided that they could throw everything into a folder under /opt, including copies of all dependencies, with no references outside of that folder, and that would be enough.
Sure, as long as you never move it from opt. UNIX devs' addiction to hard-coding fixed paths is difficult to break.
> And it might have been enough if it wasn't for sysadmins in charge of uptime tearing their beards out over /opt constantly filling up and causing multi-product outages when it did.
Is that really a problem outside of Linux, which famously defined its base-system as "the kernel" and therefore necessitated that any such install would need to bring along all its own dependencies because it couldn't rely on anything existing except the kernel?
> Of course, today, with the price of resources having crashed through the floor, and containerization having more or less reached maturity, this is all more or less an academic, historical, moot point. But dynamically linked libraries weren't stupid, they were an artifact of an age where statically linking libraries was horrendously uneconomical and the idea of namespacing anything was enough to get laughed out of the room by people wondering how you would expect to pay for any of it.
Yet we still live with this bullshit today anyway and the Linux community especially eschews any attempt to move on. This despite some of the same people who created UNIX creating Plan 9, which rejected dynamically linking and embraced namespacing over twenty years ago!
this only happens if one isnt aren't managing applications coherently (e.g binary only packages managed haphazardly, like most linux binary packagers)
on systems where there is a clear base+add-ons distinction (e.g. source based ports), these things happen much less often - all packages are built from a coherent source tree, and read only root is often feasable because packages only install reference files into their respective '/usr/share' area and don't muck with the 'real system' in /etc - the latter being an administrative action (e.g. package installation is a separate action from service configuration / execution)
The OS should be a platform. I should be able to build an application and distribute it to users and have it just run on that platform without involving third party middlemen and regardless of other software they have. You know what OSs accomplished this? The original Macintosh, RiscOS, DOS, and (mostly) Windows 3.1. Why is it that we can't accomplish today something we did 30 years ago?
a platform for 'building your system', or a platform for 'distributing and executing binaries'?
the binary only distribution mentality comes from microcomputer culture, which postdates unix/mini culture and is rooted in non-technical users installing purchased 'application' programs from vendors rather than computer users (read: programmers) building their systems to suit their own personal data processing tasks. you highlight this in your choice of platforms 'getting this right', but really, these are two different mentalities, cultures, and modes of operation.
not saying one is superior to the other, but square peg, round hole.
and besides:
many proprietary programs for unix just ship with static executables. problem solved.
These are not mutually exclusive goals.
> the binary only distribution mentality comes from microcomputer culture, which postdates unix/mini culture and is rooted in non-technical users installing purchased 'application' programs from vendors rather than computer users (read: programmers) building their systems to suit their own personal data processing tasks.
"Non-technical". Elitist bullshit. Woz was non-technical? He was clearly a micro guy. People got work done with micros. They made things with micros. Like your glorious programmer master-race, they configured their environment into something that enhanced their lives. They bought and used software the way you import 10000 dependencies. Excel is still by far the most used programming environment in the world, with entire businesses run on tooling never touched by programmer-master-race hands.
Besides, programmers should want it this way too. Which makes more sense to you? A constantly mutating state where changes in seemingly unrelated portions can wreck havok on the whole, or a collection of immutable components which can be swapped in and out at will? A system so complicated it needs sophisticated management software and a database of conflicts and dependencies to keep it all running, or one so simple that 'copy' and 'install', and 'delete' and 'uninstall' are synonymous?
If the UNIX model is so damn great, why the fuck does Docker exist!? If development is so damn great on UNIX, why are people building and distributing dev environments in containers!?
> you highlight this in your choice of platforms 'getting this right', but really, these are two different mentalities, cultures, and modes of operation.
> not saying one is superior to the other, but square peg, round hole.
I disagree. I think one is clearly objectively superior to the other.
> many proprietary programs for unix just ship with static executables. problem solved.
Ha! Yeah. Problem solved. Even for proprietary software with a native Linux version I've almost always had better luck running the Windows version under WINE.
My main threat scenarios are hardware failures and encrypting trojans. Snapshotting (if done right) protects also against the latter.
Boot environments are very handy for updates:
* https://ramsdenj.com/2018/05/29/zedenv-zfs-boot-environment-...
* https://mwl.io/archives/2363
Also, being able to snapshot the entire system at regular intervals, show diffs between snapshots, and then perhaps do security analysis on the results might be handy.
probably everyone if/when ubuntu decides to make this the default, which they probably will, because `sudo apt install coolpoints`
I loved the flexibility of BTRFS in this regard - you could change the duplication level at any point. BTRFS seemed to abstract the storage away beautifully - just give us your drives, any combination of sizes, tell us how many copies of your data you want to keep and we'll handle the rest. It wasn't perfect, but I loved this approach, and I feel like it could go even further, eg. calculate risk of failure probabilities and then just tell it how low you want your risk. If it can't do it with your current drive capacities it could tell you what it needs to accomplish it, and then rebalance the data once you add the drives. Got an SSD? Throw it in the pool and let it optimize for cacheing. I know bcachefs has that particular aspect covered, but there's a lot of other basic features that still need to be implemented.
And there is work on it for OpenZFS: https://www.youtube.com/watch?v=Njt82e_3qVo
for zfsonlinux As far as I understand you can currently add and remove striped/mirror devices, but you cannot remove raid-z vdevs (ref: https://github.com/zfsonlinux/zfs/issues/9129) - so there is definitely work left to be done in this area.
as an aside mdadm has surprisingly fantastic features in this area and can reshape pretty much any array type to any other. i was impressed by that.
Btrfs has a bad reputation because to many people lost data, including me. I would never again use Btrfs, only ZFS.
Give it a try, it's part of the kernel and it works really well.
When it was eating my data it was also supposed to be stable. But turns out it wasn't.
If you want to use btrfs go ahead, I will not do so again.
Since then all other devices made or officially supported by Jolla are using EXT4 on LVM, including this one Xperia X running Sailfish OS I'm typing this on.
(And yeah, those issues were how I found out: unable to create files while having "4 gb of free space" - not helped by the fact that it didn't run the balancer automatically.)
when will linux people understand that UX is important if they want people to finally switch from windows?
Take a look at Apple, and how beautifully designed their OS install process is, even a retarded old granny can install/reset macOS