Hacker Newsnew | past | comments | ask | show | jobs | submitlogin
The CentOS Project Just Committed Suicide (fosspost.org)
90 points by Manozco on Dec 10, 2020 | hide | past | favorite | 61 comments


I agree that RH's move is selfish and counterproductive, but I don't really agree with the "betrayal" rhetoric, except in a very limited sense.

The core CentOS devs who were "acquired", along with the people from other RHEL-clone projects like Scientific Linux who merged with CentOS, are being stiffed here; but arguably they should have known they were making this kind of a risk when they agreed to be acquired with a governance structure that allowed RH to overrule whatever the CentOS Board did.

The users of CentOS have no right to be upset whatsoever. They've been benefitting from millions of man-hours of engineering time RedHat have spent making RHEL without paying a penny for it. Granted this all benefitted RedHat, but that's beside the point -- if you don't have a two-way relationship with someone, you don't have any right to be upset when they change direction.

Pre-2014, all of those users should have recognized that the group making CentOS might disappear at any time, leaving them having to switch away from their current setup; post-2014, all of those users should have recognized that RedHat might change their attitude at any time, leaving them having to switch away from their current setup. They chose to take that risk because it was free. Now it's time to pay the piper.


The "betrayal" sentiment comes from Red Hat reneging on their commitment to support CentOS 8 in particular.

Prior to this announcement, Red Hat promised to support CentOS 8 until 2029 (the same life cycle as RHEL 8), and I have no doubt that many organizations made the decision to invest in and deploy CentOS 8 based on that promised lifetime. With CentOS 8 effectively dead at the end of 2021, that leaves those organizations less than a year to find another solution. The fact that CentOS 7 will remain supported until 2024 (which was Red Hat's initial commitment) makes this situation even more awkward for the engineers that advocated for CentOS 8.

Now, you could argue that these organizations have been free-loading off of Red Hat and got what they paid for, but it's understandable why people are upset. If Red Hat had made this decision earlier (i.e., making CentOS 7 the last version), or made good on their promise to support CentOS 8, I doubt there would be anywhere near as much outrage.


> ...it's understandable why people are upset. If Red Hat had made this decision earlier (i.e., making CentOS 7 the last version), or made good on their promise to support CentOS 8, I doubt there would be anywhere near as much outrage.

Yes, they certainly broke their public commitment; I totally agree that they should have either announced that there would be no CentOS 8 from the beginning, or announced that CentOS 8 would be the last such beast, and after that CentOS would be stream-only. And it would be totally reasonable for the rest of us to take RedHat's other public commitments with a grain of salt from now on.

But that doesn't rise to the level of "betrayal" to me. To me, betrayal implies a stronger sense of obligation -- generally from mutual dependence. I just don't think that organizations have that much obligation to people who are consuming their loss-leader. (See also Travis-ci.)


Where is this commitment written?


> Where is this commitment written?

In the public documentation describing CentOS's life cycle, CentOS 8 was described as being supported until May 31st, 2029. Upon the announcement of CentOS's change of direction, this was changed to December 31st, 2021.

You can see the commit diff here: https://wiki.centos.org/About/Product?action=diff&rev1=123&r...


Not on redhat.com or done by a red hat employee? I see.


The CentOS project and all of its assets (including its web site) are owned and managed by Red Hat, and the person that made the commit changing the documented end-of-life date -- Rich Bowen -- is a Red Hat employee with an @redhat.com email address.


The person who changed the date to 2021 is @redhat.com. Who put the 2029 date on the wiki?


Citation for red hat’s commitment to support Centos 8 until 2029 please? One that is from Red Hat not a CentOS community member, please.


> Red Hat promised

If a corporation goes back on their 'word', who else can you believe? Honestly, what is this world coming to.

> it's understandable why people are upset

... because they have to pay for a product they use. very understandable, yes.

so what?


I have a better one for you. Don’t start an open source project if you intend to make people pay for it. Go ahead and sell your support but don’t pull a bait and switch on people who develop their business models around the open source community. Plus, it makes the rest of the open source community look bad when someone pulls shit like this.


Your bad decision to base your business on something of value that is being given for free is no one else’s problem than your own.


Why don’t you tell that to everyone running Apache.


Not sure if I agree with the conclusion. maybe this is an issue, maybe times have changed. From my personal experience the 'stable' releases are just to slow to keep up with the demands on frequency of updates.

To illustrate what I mean, and this is ofcourse very subjective and dependent on my use case etc: I work on and manage hosting for RoR applications. I have a linux laptop I cannot develop on because I cannot install a recent version of ruby. Yes I can compile it manually or whatever, but that defeats the point of having distros etc.

On our hosting platform we use AWS linux, which is based of Centos. The latest version of Ruby on AWS linux is 7 years old. Our app probably wouldn't even run on it.

So we use containers. Special custom ruby images with recent versions build in a stable debian base. Now I have the problem that a.o nginx and nodejs have no recent versions available; versions that we either need for features or just because random dependecies.

So maybe the market for stable == 5+ years behind is simply not that viable anymore. Other distro's fill that demand and now centos can offer a distro for situations where you need a bit more recent versions.


> From my personal experience the 'stable' releases are just to slow to keep up with the demands on frequency of updates.

You are missing the point that it's a desired feature of CentOS distro that is targeted at servers - stability is valued more over running the latest versions of some software that isn't tested over a long period of time and may or may not have bugs.

Most people use CentOS in servers because it provided a very easy migration "upgrade" in the future to Red Hat Enterprise Linux (CentOS is a near clone of RHEL, without the Red Hat branding). Now nobody will consider it anymore because the new CentOS is not tested for stability (using tried and tested bug-free softwares) and won't provide an easy migration path to RHEL anymore.


RHEL is used for its stability. I also would like to see more updates. I think it would be better to introduce a RHEL Stream in addition to “basic” RHEL instead of killing CentOS.


Not everything chases the latest and greatest though. If you run containers anyway, wouldn't you want the host running those containers to remain stable and predictable? Would you gain value from running your databases on rolling-release distributions?

It may be that your application benefits greatly from always running the latest versions of whatever technologies you use, but there are tons of use-cases for a more slow-moving platform as well.


> If you run containers anyway, wouldn't you want the host running those containers to remain stable and predictable?

For this use, I'd say one should use a disto focused only on running containers. Everything else is overkill and subject to "feature" creep on the server.

There is also the issue that all the container stuff is still a moving target feature-wise.


> The latest version of Ruby on AWS linux is 7 years old.

On RHEL 6 and 7, Software Collections are supposed to help with this: https://developers.redhat.com/products/softwarecollections/h...

You can install rh-ruby26 which doesn't sound too bad to me.

On RHEL 8 there are modules/appstreams (according to https://access.redhat.com/support/policy/updates/rhel8-app-s..., you can use the ruby 2.7 appstream today which will be maintained until March 2023).

Red Hat also produce container images for those who want to go that way. Of course you can use any container image you want... I'm only mentioning this because the ubi8/nginx-118 and ubi8/nodejs-14 container images seem pretty up to date to me... ;)


The idea is that the server base is rock-stable, and then you judiciously install newer versions of software that is central to you and your business (or activities).

So you would install a newer Ruby, maybe even JRuby?, and a newer Rails, but you're probably not interested in time-consumingly maintaining postfix. Or systemd. Or gcc. Or the GNU userland.


> On our hosting platform we use AWS linux, which is based of Centos. The latest version of Ruby on AWS linux is 7 years old. Our app probably wouldn't even run on it.

Are you using Amazon Linux extras? That’s what I use for current Python releases and it seems to have more recent Ruby versions:

https://aws.amazon.com/premiumsupport/knowledge-center/ec2-i...


If you're developing you should really use something like asdf or at the very least RVM. Depending on system packages as a developer seems kind of self-defeating.


Not saying this fits your problem but JRuby is great, you can just package it with your application and never have to worry about version problems of Ruby anymore. (You do have to have Java, but that's very easy.)


Building an SRPM is literally only one command.


I'm not sure what the issue is here, TBH. CentOS Stream is not a pure 'rolling' distro; it's still tagged by CentOS major release, i.e. there will be a CentOS Stream 8, CentOS Stream 9, etc. The "rolling" logic applies to minor updates within a CentOS major release, but even before this, it's not like RHEL or CentOS were publicly releasing backported updates to earlier minor releases, so practically speaking you were forced to be on the latest minor release anyway. People are correct to point out that this is somewhat of a trade-off in stability but it was always there. The one thing that's new with Stream is that there will be a faster release process for these updates, and that's a very good thing.


No one is installing something self described as the "development branch" on their vital infrastructure.


We're talking about the "development branch" for minor, bugfix point releases. It's not going to include major or disruptive changes.


My company got burned badly by this move but to play a little devil's advocate: there's no such thing as free lunch as it turns out sooner or later. Free/Open source != free as in beer. You either pay it in financial support or sharing of development and maintenance of such projects. Sooner or later.


You can always choose BSDs. NetBSD foundation is NGO running since 26 years.


Sure there is, the Debian project. I never understood the whole reverence for RHEL, when an alternative has lived alongside it for the whole time and longer. I've been using Debian since '97 or '98, and with one (glaring) exception, stable upgrade paths were always trivial when compared to Centos "nuke and rebuild" approach, and you get 5 years of support, which is really quite decent.

Of course, Debian will not sell you support but you have places that will. I've never worked with a truly giant fleet, so I'm sure I'm missing something, but I also feel very fortunate at the moment to have always had the choice to work with Debian both personally and professionally.


I agree. But I rather pay a small fee for CentOS without support than a hefty fee for RHEL with support.


> Free/Open source != free as in beer. You either pay it in financial support or sharing of development and maintenance of such projects. Sooner or later.

It can if you want it to. Just like with free beer, if you doN't want to drink that beer you have to go find other beer.


There's a possibility that this isn't the big deal people think it could be. It all depends on the day to day stability of Stream. Red Hat distros already have a testing repository channel for example. If stream updates are QAed through something like this the resulting platform may well be stable enough for many use cases. It would be no different to enabling the 7x/8x repo in the current releases.

In many respects I'd rather have a slow trickle of updates than the current flood of each point release. We've had lots of experience of dealing with all those changes landing at the same time breaking things.


> There's a possibility that this isn't the big deal people think it could be. It all depends on the day to day stability of Stream.

CentOS's key attribute, and what separated it from other enterprise-focused Linux distributions, was that it is (well... was) a binary-compatible clone of RHEL, and was supported for as long as its RHEL counterpart. This allowed the user to use CentOS as a free drop-in replacement when the requirements called for RHEL (e.g., when running proprietary enterprise software that only supports RHEL).

CentOS Streams is not a binary-compatible clone of RHEL, which makes it unsuitable for people who need that specific feature.

CentOS Streams may be perfectly reliable (note that this isn't the same as being stable), but there are already many reliable and well-established Linux distributions to choose from if RHEL compatibility and length of support isn't important, and few reasons remaining to choose CentOS.


How is CentOS Stream not a binary compatible distro with RHEL. Technically, CentOS was NEVER binary compatible with RHEL, it's a re-compile, but EPEL works with it. The same is true with Stream. No change there.

The change is CS Stream is literally the same change our (Red Hat) customers have been absorbing for years.

CentOS users have this perception that they were getting stability by being behind paying RHEL users. Think about how ridiculous that logic is.

RHEL users weren't being bombarded by ridiculous instability. RHEL Betas were never that unstable and besides Stream literally passes the RHEL hatting tests.

See more #6/#7 here: http://crunchtools.com/before-you-get-mad-about-the-centos-s...

So many people were consuming CentOS without a fuzzy clue to how RHEL works. Makes it all the more frautrsting for people who get a paycheck from RHEL.


I think this is most likely to be the case, but I have to admit that it still does worry me!

But with CentOS 8 slated to continue in line with RHEL 8 support dates, at least there's plenty of time to see how the stream side of things goes, and to test it thoroughly.


Nuts - ignore me, the FAQ seems to have been tweaked since I first read it, could have sworn it said that before...


Wait so is CentOS8 effectively EOLing within a year?


Yes. CentOS 8 EOL is now the end of 2021: https://centos.org/distro-faq/#q2-what-about-the-other-relea...


OMG


Yes it is.


These stable packages just postpone the inevitable technological debt and when you have to pay the debt, it's too late.

I really hope these stable OS just die. Some debt could be avoided if RHEL/CentOS didn't exist.


Being incredibly boring and old ends up being a beautifully stable foundation to build on. The idea is to pay the debt as an organization on a fixed schedule that allows for years of stability in between. For larger organizations, this can be far less work and less strife than a rolling upgrade would, as the resources to test such upgrades are often unfortunately limited and expensive.


Perfectly well-said. Some developers wants the greatest and latest, but they forgot the operational costs can be higher (see: cloud and its tendency to cost more than others). At the end of the day, there are companies who needs to standardize on a system for a long time and CentOS delivers it just fine.


I foresee a fork. Nothing stops concerned party to create and maintain their own distro from RHEL source.


Oracle Linux already builds from RHEL sources, so that could be an option.

Edit: There's also Rocky Linux already : https://news.itsfoss.com/rocky-linux-announcement/


Nothing other than it being a huge cost and commitment for no actual gain.


rockylinux.org


Opensuse, Debian and Ubuntu live on, of course.


This. If you need stability, nothing beats Debian.


So, it was my understanding that Fedora was the beta/testing upstream distro for RHEL. Features and development would go from Fedora -> RHEL -> CentOS, previously. With CentOS moving upstream of RHEL, it seems a bit strange doesn't the role overlap with where Fedora already is?


It is still. Fedora 34 is currently being branched into RHEL 9 Alpha. In fact I spent last week unifying the spec files of my packages so they are the same in Fedora and RHEL 9 (as far as this is possible).

However the problem has always been that between RHEL being branched from Fedora and the first RHEL GA being released there was a gap - a really huge gap actually of about 18 months or more. During this time we released to the public one alpha, and one beta to partners, but essentially development was closed off to anyone outside Red Hat. During this time we are writing documentation, QA-ing extensively, fixing bugs found by partners. Plus partners and ISVs actually ask for a 6 month delay before GA which they spend porting their software and device drivers.

CentOS Stream is probably a poor name, but it fills that huge gap. Now CentOS Stream releases between Fedora and RHEL, and in front of every RHEL minor release, will be available to everyone, and we'll be accepting contributions from everyone too.

Dropping CentOS Linux (the rebuild) is kind of separate from all this. It looks like it's going to continue as a community project as it was before 2014 (https://rockylinux.org/) which may be better in the end because the community will be more aligned to the success of the rebuild.

Disclosure: I work at Red Hat.


well as it says in the article "[with CentOS] you basically get a free Red Hat Enterprise Linux with same packages and same updates." - I assume that don't sit well with IBM, the owner of Red Hat the owner of CentOS.


I agree. We will see if a new community wants to copy CentOS and maintain it.


Original author of CentOS just created "Rocky Linux"


I know at least 3 people who are right now looking at what it will take to migrate everything they run centos-related, to debian stable.


Not being a sys admin or that. How much work is it to change distros for say a mid size tech set up?


It really depends how specific whatever software/application is to the unique layout and structure of centos vs debian.

If it's some common LAMP stack thing duplicating exactly the same thing on debian+nginx(or apache2)+php7.3+mariadb could be trivially easy.

For organizations that have things like ansible set up to auto provision things where the configuration files live on a centos system, obviously locations and naming will need to be changed.

there are also certain weird (usually closed source binary) applications shipped by software vendors for specific vertical industries, which ONLY exist in a form for RHEL or CentOS. It'll be interesting to see how that works out. The vendor won't support running it on debian or ubuntu server or anything else.

I would say that out of all the possible distros of Linux, RHEL is by far the one that I see with the most rare, weird, expensive paid software that gets developed for it, due to its extensive use in government and fortune 100 type environments. In which case I don't think much will really change since the same customers will go right on buying RHEL paid yearly licenses.


if you have 100 machines you need to install 100 new OS's which just by itself is a bunch of work.

The actual install of linux is generally pretty straight forward but to do it minimum downtime you need spare boxes to build with the new distro (preferably 1 for each old box, but you can get away with less, its just takes more time), install your software on, and then test. You also need sysadmins and testers to do the work, and developers to fix any issues (normally around library versions etc). Then you have the risk that something was missed in the install & testing and causes issues in prod a new distro will not be 100% bug compatible with your old one so you will find problems.

And thats assuming you have the skills to install the new distro, how to set it up as required, how to reinstall the software (its amazing how much in house software doesn't have install docs or even source code), and the skills to run it 24/7. If not factor in training and self-learning time for everyone.

Its all doable, but its probably months of work and a bunch of risk just to get back to where you were on a new distro, which makes no difference to your customers at all.

Theres a reason companies tend not to switch very often.


absolutely huge pain in the ass. The issue isn't typically around installing a new linux distro; that works fine, these days. it's usually getting all your in-house code and processes to work properly with the new environment that's the nightmare. you'll be handling weird bug reports related to the migration for months.

which is in no way redhat's, or centos's fault.


And to think my team is migrating to CentOS 8 as we speak...




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: