The article has a great example of how competition is healthy even in the open source space, with the Btrfs authors wanting Bcachefs merged into Linux Mainline. A lot of people that got burned with Btrfs will use Bcachefs instead when it lands in mainline, but having both in the kernel will make it much easier to get kernel interface changes and tooling that benefits both Btrfs and Bcachefs.
How did they get burned? (I'm thinking of switching to Btrfs on my personal laptop (running Arch), mainly for the snapshots, subvolumes, checksums+deduplication, etc.)
I lost a root filesystem twice, on OpenSUSE in a one-disk system (i.e. what should have been the most stable well tested configuration), around 2018-19 IIRC. It now appears most likely that the system ran out of free space (suse ships snapper set to snapshot on every package operation, and the partition was smallish), and that apparently triggered corruption that ... at least, I couldn't figure out how to fix. Power loss may have been involved. Details fade from my memory, but the violent aversion to ever trusting it remains.
It is actually wise to shrink a btrfs filesystem on its partition by a few megabytes, so some expansion room is available in an emergency.
There are very few filesystems that can shrink like this:
btrfs filesystem resize -1G /mount
If the "btrfs fi df" command shows any of the data/system/metadata elements close to limits, it is time to take action. They can all push you into read-only mode.
I lost a btrfs-raid5-like NAS out of the blue (after 3-4 months of using it, I didn't even write a lot on it):
the filesystem was always cleanly unmounted, I didn't do any crazy experiments with it, it had plenty (>40%, hundreds of GiBs) of free disk space; on one day I noticed that accessing some files was slower than usual, the noise of the HDDs had a weird pattern and from there it went downhill (at that time there was no fsck utility so I didn't even know what to do - if I'm not mistaken, BUT I MIGHT, the general opinion was that a fsck-util was not even needed? but again, I might have just dreamed this) and finally stopped working with a huge amount of error messages in the kernel logs => I had to reformat everything (I went back to mdadm-raid5) and restore from the secondary NAS' backup.
In my opinion there were 2 main things that contributed to the bad reputation of the FS:
1)
In those days there was a looot of hype about btrfs (as well for me it was a dream come true) which is the reason why I decided to use it.
As far as I remember I didn't have to enable any special "experimental" feature in the kernel to use it => that gave me the impression that "it works under normal conditions, the DEVs are probably being a bit conservative for corner cases".
2)
All functionalities (e.g. raid5 with rebalancing in my case) were all available from day 1 (at least when I started using it) so if something's available I tend to think that it works (I mean, from a user's point of view why make something available that does not work?) - but no, apparently many people had many problems with many different configurations/usage patterns.
-
As others have written I hope that Bcachefs will primarily focus on reliability and only secondarily on performance & features, but I admit that that's difficult to balance :). And I hope that the fsck-utility will be available from day1 (at least with a reduced functionality). And I hope that "experimental" features will need some kind of very explicit command line option (e.g. "-might_break_everything") to use them through userland utils.
Go Kent!!! :) I'm since a while one of your Patreon follower - If you manage to create something usable I'll print the pic shown on LWM and I'll hang it up in my PC-room, maybe I'll even take my twitter account out of cryosleep to tell Musk that he should use Bcachefs as FS for his Teslas :))
"The RAID56 feature provides striping and parity over several devices, same as the traditional RAID5/6. There are some implementation and design deficiencies that make it unreliable for some corner cases and the feature should not be used in production, only for evaluation or testing. The power failure safety for metadata with RAID56 is not 100%."
Many people used btrfs while it was still experimental, and lost data.
Attempting to create a btrfs filesystem returned a prompt with this warning (until 2013):
Btrfs is a new filesystem with extents, writable snapshotting,
support for multiple devices and many more features.
Btrfs is highly experimental, and THE DISK FORMAT IS NOT YET
FINALIZED. You should say N here unless you are interested in
testing Btrfs with non-critical data.
That prompt was removed when things stabilized, and Fedora's use of btrfs as the default root has not been accompanied by complaints of excessive filesystem corruption (although there are still many rough edges and unfinished features).
Since Btrfs was arguably available in the kernel too soon. Some people who should have avoided a new filesystem for lots of reasons ranging from missing features that were in the plan but not implemented yet or in the earliest days not yet stable... many of these early experimenters developed and fostered a vocal belief that Btrfs was "bad" (there's too much to sum up honestly, you could write an entire decent length blog post on the history of Btrfs, its reception and its criticism) and this group has taken a long time to accept that Btrfs may be ok now. Theres still people with the perception that its not as good or ready enough and should not be used.
> We prioritize robustness and reliability over features and hype: we make every effort to ensure you won't lose data.
This is exactly what btrfs failed at, this is not just an issue with the code itself, but also with the developers. Why should i trust feature-oriented developers with writing my infrastructure code, where reliability and robustness are top priority?
There's a list of features and readiness at https://btrfs.wiki.kernel.org/index.php/Status - there were people who didn't check it and used unstable features and experienced issues. This is partially on the project for not making it more explicit.
The second is that there are people who read that, see the unstable raid5/6, and are vocal about btrfs being overall broken / not ready rather than accepting that feature just not being available yet.
well, FAT16/FAT32/EXT2 are not journalled, so making them lose data in a giant variety of ways is a trivial affair. You might want a higher bar than that.
That's the point: if file systems that basic would NOT lose data, then it is unacceptable if another file system does.
Maybe to put it differently: new file systems should offer improvements wrt reliability for stable features of the file systems of the previous generation; regressions are unacceptable.
Yeah, when you get down to it the opinion really boils down to "the file system is broken iff it doesn't lose data for the use-cases I personally use it for."
We've been waiting so long for proper software RAID support that ZFS got RAID-Z expansion before it happened. I no longer have any reason to use btrfs.
I've been running full Btrfs on my workstation since December 2021 with 0 issues (kernel 5.10). It survived multiple hard reboots and power offs, Ryzen 5000 CPUs seems to lose power every now and then on Linux (it's a known issue for years now that no one has figured out yet).
Having compression is great. compsize reports I've saved 33GB on my /home subvolume so far (https://github.com/kilobyte/compsize). I don't use snapshots, I just backup the whole thing.
Obviously this doesn't mean that in the future Btrfs won't eat my data, but so far it hasn't and I'm happy.
I lost not one but TWO entire volumes to it in the past few years. Just one day they wouldn't boot. I wasn't using encryption and I wasn't using any RAID feature, just compression. The people who think BTRFS is a viable substitute for ZFS have just gotten lucky with not getting burned.
There's also the issue with BTRFS performance degrading hugely with any significant fragmentation, to the point that "autodefrag" has to be a feature, which of course also adds SSD wear...
In the early days after Fedora made it available, I lost my root twice after a kernel panic and later after a hard power reset.
EDIT: in comparison, I've been using xfs for almost a decade now with numerous hard power offs and kernel panics. I've only ever lost select files that were open for write at the time of restart, and not the entire filesystem.
I wonder how Fedora started using btrfs as the default for their desktop version when RedHat had deprecated it years ago.
Never used btrfs but I see corruption reports in here everytime I read about it, and it's been in development for over a decade and this thing needs to be put to rest before it eats more people's data.
Hopefully bcachefs becomes feature parity and stability against zfs to provide a modern filesystem to wider Linux audience at last.
Funny to see we're still using ext4 as the default filesystem which is nothing fancy forever.
> I wonder how Fedora started using btrfs as the default for their desktop version when RedHat had deprecated it years ago.
Because the Fedora community wanted to, basically. They're not obligated to go along with whatever Red Hat wants to do, although there is naturally some alignment there. Plus having a different default filesystem really isn't that big of a deal.
Well, I always hear about data loss incidents whenever btrfs shows up in HN comments, so I thought making it a default in Fedora desktop sounded like a big deal as there must've been a good reason on why RH dropped it.
Maybe it's always the same stories. I haven't read of any happening in the past couple years. Of the two I've read in this thread, one happened ages ago, the other involves raid5 which AFAIK is not recommended for production use. Also, kernel bugs tend to get fixed, especially now that btrfs is recommended by a few distros.
I know nothing about the reasons it was dropped (apart from what I've read on HN, ironically) but keep in mind this was years ago, when it was still deeply, deeply troubled and there was a lot of churn going on. "Churn" and "stable" kernels like RHEL don't mix well.
I've been running in danger mode of RAID6 on a 15 drive NAS here at my home for the better part of 5 years now, and I have had zero issues. The server is also running Arch and I run package updates every two weeks or so. Maybe I am just lucky.
My Linux laptop, running OpenSuse Tumbleweed, got its root fs into a state where it would hang when trying to mount RW. I was able to boot off of a rescue image, mount it RO, and copy my data to another machine. None of the repair suggestions were able to get the fs into a mountable state, I had to reinstall it and copy my data back.
Every VM at work that uses btrfs gets into a state where some maintenance process (rebalancing, I think - I'm not one of the Linux admins) will consume all CPU in the VM and prevent it from doing any work. The only thing that varies is how long it takes - it seems to be related to how much write activity there is on the fs, VMs with heavier write activity get into this state faster. This has led to outages and the Linux vendor was unable to resolve this issue. We've since removed all traces of btrfs from our data center.
If you used Docker on your machine, this was a bug in the remount code (they forgot to re-trigger the qgroup scan if it was interrupted when you did an rw-remount) which I fixed several years ago[1]. I don't know what that will do for your confidence in btrfs, but hopefully it'll give you some closure.
The incident with my Linux laptop was definitely not Docker-related, but the description of the issue sounds like it could still apply.
We didn't use Docker at work, at least not on the VMs that I was affected by. This one was a gradual degradation over time - the maintenance process would run fine, but after a while it would start hanging the machine. It really felt like the fs was only good for some amount of writes before the maintenance process got overwhelmed and would hang the VM whenever it ran.
I'm happy to see that things are getting better, but the reputational damage will take a long time to repair.
All that being said, I still use btrfs on my Linux laptop, but I've also become more conscientious about backing it up.
I was told a while ago that rebalancing could get into pathological cases fairly easily (on openSUSE we used to run a rebalance in a cron job and it turns out this would grind machines to a halt). I still use btrfs and haven't had this issue for several years so maybe the issue has been resolved? I'm not sure what the resolution might have been.
BtrFS is the only filesystem I've ever lost data with, and I've used a lot of filesystems over the decades (UFS, VxFS, JFS, XFS, EXT2/3/4, ZFS).
I have a comparison of features between BtrFS, ZFS, and EXT4+LVM at [0], and as I note at the top of that post I was looking to move to BtrFS. I did so and lost data.
The main thing I found btrfs was good for was testing the restore process of my backups. I was using it in supposedly known-good configurations (RAID 1 or RAID 10) and it consistently destroyed data.
The fact that it turned out that the scrub function spent years corrupting data by mistake was only one of the many flaws with it.
I'd recommend listening to some of the 2.5 Admins podcasts. There are a few episodes (earlier on I think) where one of the hosts lists a number of issues with how it is implemented and basically wouldn't recommend it for anything, even in trivial set ups with a single drive there are far better alternative's.
How well does bcachefs handle VM or DB workloads? Those have always btrfs's achilles heel unless you disable copy-on-write and checksumming, whereas zfs handles them pretty well if you set the recordsize correctly.
Mainline within next 6 months?
Man, that would be pretty exciting.
I will probably wait longer, but always glad to hear about development of bcachefs :D
Having it stable in mainline in 3-4 would also be a big achievement. The filesystem part in Linux is kind of stagnating:
* ZFS will forever be problematic because of licensing reasons
* BTRFS stability is widely untrusted after their problems many years ago
* ext4 and xfs don't have data checksumming
* To get the same benefits of ZFS you currently need to stack a lot of device mapper plugins
Sparc, Solaris, and various zfs based products that Oracle sells still have a shelf life of a decade, give or take. There is little reason for them to do anything.
Someone on LWN seems to think that Solaris/SPARC are essentially EOL already for Oracle and the scales are tipping towards having ZFS in Linux mainline.
Yeah, I use ZFS, and with some distros its... quite painful to get root-on-zfs,
On the other hand, I use NixOS, and ZFS is nearly as easy to set up as any other filesystem, and afterwards its just bliss.
How is it a problem from the user's point of view?
If you're on Ubuntu, even if you have ext4 on root, all you do is install a single package and allocate an empty partition for zfs and you're good to go and it's stupid to miss out on what zfs offers by sticking to ext4 forever.
Flexible space allocation between volumes, instant snapshotting, transparent compression and backing up the entire filesystem to a remote server is already available if you choose to use zfs.
Far better than using btrfs with endless report of corruption.
I admit it took me longer than it should to notice that this is not about some "BCA chefs" but rather "B-Cache-FS" ... and I quite closely follow Linux filesystems...
Writing it as BCacheFS would have helped, but CamelCase is not very popular these days it seems
CoW (copy-on-write), deduplication of data blocks, replicas (RAID equivalent), file system snapshots and multiple, tiered caching layers, plus several other features.
On a sidenote, I integrated bcachefs as a vmx driver into VMWare ESXi for a CI/CD build server a few years back. The build system ran in VMs and on containers, but the non-essential target directories sat on bcachefs volumes with the caching layer directed first at RAM, then at SSD, then finally at the HDD. Managing all the Unity3D and Unreal caches was amazingly fast across dozens of different SKUs of what the same project.
From my project write-up on my LinkedIn:
Bcache-like caching layer for VMWare ESXi
Reduce latency and read/write delay even if using SSD as your storage.
Written in C and inserting itself as a storage tier into VMWare ESXi to handle read & write storage requests this caching system accelerates all accesses to the underlying backing store.
Can work in both write-back and write-through caching modes. (Native C kernel device driver)
The thing that I’m most looking forward to: the ability to have multiple drives of different sizes, and expand the array with new oddly-sized drives (great for a homelab NAS built of spare parts; btrfs can do this, but zfs can't), combined with being able to set some drives as caches (eg having the filesystem automatically store frequently-read data on an SSD and rarely-read data on HDD; zfs can do this, btrfs can't)
There's a ton of features, but really the most important one is that CoW filesystems tend to extend checksum protection to the actual data you're storing on them. Journaling filesystems generally only protect themselves and not the data.
With dm-integrity you either have a small hole of a few milliseconds (bitmap mode) or write all data twice (journal mode). When integrating everything into a cow file system you can sidestep the issue, as you basically journal the data through cow anyway.
The only small advantage I can think of is that the filesystem knows which parts of the drive are being used, so it will not try to recover stale/unused blocks. Same goes for writing - dm-integrity needs to initialise the whole drive and update every single write, but the fs can issue some big TRIMs instead. (Or does integrity know how to trim these days? (Apparently it depends on the usage mode https://www.kernel.org/doc/html/latest/admin-guide/device-ma...))
I don't know about bcachefs' capability on this but with zfs, you can do an instant snapshot of MySQL/PostgreSQL data directory and call it a backup instead of fighting the tough fight that is database backup with their dump utilities taking a good amount of space and time and not easy to do incremental backups.
I really want a filesystem with snapshots as the foundation of a better backup system. Bcachefs has other useful features like CoW and disk pooling (in fact snapshots weren't even on the table until recently), but in my mind snapshots are a must-have feature these days, like journaling was 20 years ago.
Unfortunately, this has been just around the corner on Linux for over a decade now, and the two filesystems that promised to deliver it are unlikely to reach mainstream support on Linux. ZFS has licensing issues, and uses too much memory for a desktop system. BTRFS tried to do too many things, and has had too many reliability issues for me to trust it.
Licensing issue is only for distribution. You're free to use it without breaching the license. Not sure why people won't just use it.
How is zfs using too much memory?
zfs can run on a 2GB server (with some swap). Any laptop would have enough memory to run it. You might want to change "zfs_arc_max" as zfs tries to use half the memory available on a system if you don't set it and don't use deduplication.
Probably won't replace zfs, but if the recovery / failure handling story is better than btrfs, maybe we'll see that one replaced. They're going to be fairly similar in features and concepts.
If it just has all the features zfs has and none of the years of stability that zfs has... why would any current user (who is by definition ok with the licence) take the effort to switch? What's the extra benefit that zfs currently doesn't provide?
> What's the extra benefit that zfs currently doesn't provide?
Not being at the mercy of a company like Oracle is a huge plus in many ways. A huge plus for future development and risk-free adoption.
I use ZFS on a server of mine, but I am one of those paranoid people that would switch just to get away from a project that could be hamstringed at any moment if Oracle has one of its episodes again.
ZFS (hopefully) never finding its way into the mainline kernel is kind of a meta-disadvantage.
They can't just relicense OpenZFS. The only attack vector is suing Ubuntu for distributing zfs because Canonical's lawyers think that's fine but even then, you're still free to use it by adding it to the system yourself like any other distribution is doing.
Running a new filesystem is far more dangerous than your theoretical legal concerns.
They can go after end users if they manage to find a business angle where they can start charging for the system. "But that's silly, Oracle wouldn't be bothered by small businesses, right?" https://www.reddit.com/r/sysadmin/comments/d1ttzp/oracle_is_...
OpenZFS is licensed under a free software license. There's no mechanism by which they could demand anything. At best they could try to sue Canonical for CDDL violations, though the legal arguments online suggest that Oracle wouldn't have a leg to stand on here because the license incompatibility comes from the GPL side (meaning that Linux copyright holders might be able to sue Canonical but even that is a stretch since it's hard to argue OpenZFS is a derivative work of Linux).
The thread you linked is someone who was using software with a proprietary license against its terms and is being asked to pay for its usage -- obviously I think Oracle is being scummy but it's not a comparable situation at all. This would be like saying that you won't use VS Code because Microsoft once demanded that someone who was using a cracked copy of Windows pay them -- it's a complete non-sequitur.
Just being in the mainstream kernel, instead of an out-of-tree module, already gives a couple of benefits. The code will always be kept up-to-date with the internal kernel API, so you won't have a situation where the kernel is updated but the out-of-tree module fails to compile due to an API change. And if you're running a signed kernel (for SecureBoot), you don't need to have a complex setup to add an extra key and sign the module whenever it's rebuilt. Also, as others have mentioned, being in-tree means it's better integrated with the rest of the kernel; this includes making changes to core kernel code when they could help the module.
Big chunk of memory - that's not the case except if you are using deduplication. I have it running fine on a 4GB machine (10TB mirrored volume) without any issues.
What I believe gp was referring to was that zfs uses different caching system that the main system. Specifically arc vs page cache do almost the same job, but are separate and may fight for resources. Discussed in a few places, but here's an example with some behaviour summary https://www.reddit.com/r/zfs/comments/o8xqzb/zfs_on_linux_ca...
Because right now it "has no features" because it's not really widely available. No installation, no features. The proof of the pudding is in the actual shipping and installing and using it. :)
Empty threats are void, IMHO. Might take a bit of bravery to step over the fence, but if no one comes after you after a certain amount of time has passed, the fence is effectively not there. Ubuntu already did that work, though, so...
No, I mean SPL[1]. The linux devs definitely don't want a proliferation of porting interfaces mucking up the kernel tree, so unless the OpenZFS devs want to work on a Linux specific port, I don't see ZFS getting mainlined regardless of the licensing issue. CDDL certainly doesn't help to make this happen though.
Also "Empty threats are void" is a silly stance to take; betting that nobody who will own copyright to code in the CDDL is going to sue anytime in the next 80ish years would be dumb even ignoring the fact that Oracle owns copyright to at least some of that code right now, and they are quite litigious.
Didn't know about the SPL thing; interesting. So what's Ubuntu doing, just dealing with packaging it on their end?
> betting that nobody who will own copyright to code in the CDDL is going to sue anytime in the next 80ish years would be dumb even ignoring the fact that Oracle owns copyright to at least some of that code right now, and they are quite litigious.
When I wrote this, I was under the impression that US law more or less required copyrights to be regularly defended in order to maintain any "legal weight", but on Googling, it seems that while the awards in a case where a copyright was undefended until some significant time had passed, were lesser, it was still quite defensible. So I guess, fuck, that kind of sucks in a way, because it incentivizes patent trolling/hoarding/squatting, but it also makes it possible for someone without significant financial backing to benefit from a patent without having to afford good and regular legal defense.
That's one hell of a damn tradeoff.
A possibly better system would perhaps be one similar to what happens in the music biz in which a company takes a cut of any wins or profits by providing someone substantial legal representation in the defense of their copyrighted works.
Some of us will NEVER accept ZFS's licensing. The ability to remain absolutely vigilant in regards to licensing immensely helped IBM during the SCO v. IBM debacle in regards to the JFS file system. We have learned this E-X-A-C-T lesson before.
Additionally, since the code originates from Oracle, many of us progress from being vigilant to outright paranoid. Oracle has never been forgiving, warm, soft, nor cuddly. Hell, even Torvalds himself wants Larry Ellison to personally sign off on ever including ZFS into the Linux kernel [0]
Oracle could easily fix the ZFS licensing situation by releasing CDDLv2 resolving the GPL incompatibility. Then via the CDDLv1 auto-upgrade clause, most of OpenZFS could be made compatible with the GPL. Some parts would have to be relicensed to no longer opt out of the auto-upgrade clause or rewritten from scratch to enable the auto-upgrade clause. All other CDDLv1+ software would be made GPL compatible at the same time. After that OpenZFS could be included in Linux. Some more discussion about that on LWN. Does anyone know someone at Oracle that could get the ball rolling on CDDLv2?
OpenSSL could change the license on most of the initial code base. AFAIK, ZFS is the exact opposite, with the majority of the core code having been owned by Sun and nobody can change its license now but oracle. I'm pretty sure that you're talking about a nearly complete rewrite.
Given Oracle's track record, I advise extreme caution on behalf of the mainline kernel with respect to code that Oracle could claim to have developed.
I'm less sympathetic to sour grapes about ZFS. Kernel developers including Torvalds have made false negative remarks about ZFS vs Btrfs; the SFC deployed implausible legal theories made veiled threats against Ubuntu for maintaining a patch to Linux at their own risk allowing OpenZFS to be used.
I'm actually curious; did anybody ever offer an explanation for why Nvidia can ship a binary driver with a tiny shim and not get any blowback while ZFS shipping an open source driver with a small GPL shim gets everyone up in arms?
> It's just that I'm so damn tired of this whole thing. I'm tired of people thinking they have a right to violate my copyright all the time. I'm tired of people and companies somehow treating our license in ways that are blatantly wrong and feeling fine about it. Because we are a loose band of a lot of individuals, and not a company or legal entity, it seems to give companies the chutzpah to feel that they can get away with violating our license.
The only concern is the distribution and you would mean you don't accept the fact that it can't go in the kernel or be released as part of the distro installer?
It's not hard to download zfs separately and use it like any distro is doing without having any licensing problems.
