329: Outrage Alert: Red Hat Changes CentOS Again!

2 Likes

Thanks, @MichaelTunnell , @dasgeek and @jill_linuxgirl for clarifying about all the drama around Redhat that was appearing on the net. Having been part of this forum since its inception, my guess was here would be the place to actually be correctly informed :+1:

I’ve had a lot going on so am around less than usual, so looks like I’ll be missing out on the discord forum click-bait fun but I’ll be catching up on episodes when I can :slight_smile:

3 Likes

Regarding the Red Hat kerfuffle, and the Destination Linux coverage: a lot of stuff was said that I disagree with, but is debatable (like the “parasites” language, that I really didn’t like, or how similar this “closing” is to other Linux vendors’ business models), but here’s the two main issues that you glossed over (“oh, it’s not a big deal”) or completely ignored:

  1. The GPL requires that if you distribute GPL code, you “must license it on the same terms to all 3rd parties”, which means if pay Red Hat and get access to the GPL source code of software disturbed in RHEL, the license allows me to compile it, bundle it and redistribute it without any limitation except the GPL - but the Red Hat customer licence agreement says the opposite: the agreement I must agree to, to get access to the GPL source, says that I must not distribute the software I get from RHEL, and puts all sorts of limitations that contradict the actual software copyright license - that Red Hat agreed to when they started distributing it in RHEL. Most of software in RHEL isn’t made by Red Hat and Red Hat are now breaching the terms under which RHEL was created.

  2. There’s a lot of value - for Red Hat - in various people in the world being able to test, deploy, and support applications on RHEL compatible systems: all kind of third party software developers make software that is available and supported on RHEL installations. These developers used to be able to test their products easily to be RHEL compatible, and in return their customers would buy RHEL licences for using the products on. Now it is much harder, as CentOS stream isn’t compatible with RHEL and the tinkering that Ryan suggested developers can do can never result in 100% assured compatibility. The 16 seats dev licence is a joke - you have to install licenses, you can’t really use automated provisioning (you need an uber expensive “cloud license” for that) and if you need more than 16 seats (as common in many kubernetes setups) - then you can’t even use that. Here’s one such developer explaining why he is giving up on offering RHEL support to his customers: Huge Open Source Drama - YouTube

Is Red Hat no longer “open source”? No - there’s still a lot of good open source work done in Red Hat, but RHEL is no longer open source: it does not allow it’s users to copy, modify and distribute the source.

I feel most of the coverage in the episode was a long the lines of “Red Hat had been so good to us, let’s just trust whatever they say and never criticise them”, which is disappointing because I really expected better from the Destination Linux crew.

2 Likes

This reminds of the transition to CentOS Stream. There was so much information and so many perspectives I was just grateful for anyone willing to put in the work to talk about it reasonably so I could weigh it against everything else I was able to learn and verify.

TuxDigital thank you, @guss77 thank you and thank you to everyone else figuring this out and sharing it so we can work this out together. Appreciate it.

2 Likes

Sometimes decisions in FOSS communities are difficult to understand, but I have appreciated a variety of voices who are shedding light on the ramifications of Red Hat’s decision

I found DL’s hosts @MichaelTunnell , @dasgeek and @jill_linuxgirl made a good point that Red Hat does not have an obligation to run public servers that make available RHEL source code. Red Hat does have the obligation, which they are doing, to run servers that make RHEL source code available to RHEL customers.

I would love to hear license lawyers discuss point #1 made by @guss77. How the GPL and RHEL license interact is beyond my current understanding.

Thank you, @guss77, for making point #2. This is a point that I have heard those who work in the enterprise space or develop applications for enterprise companies make. Having a “binary compatible” distribution available to develop, test, and certify compatibility with helps the Red Hat ecosystem as a whole. This enables developers to develop enterprise applications that enterprises can then run on RHEL servers knowing that they have support for their whole stack which would be a win for enterprise customers, application developers, and Red Hat making money from RHEL licensed servers in enterprise companies.

I thought I heard one of the DL hosts make the comment that Red Hat is simply doing what other open source companies are doing like Canonical with Ubuntu Pro, and Suse. But I am not sure this is true. When Canonical releases an LTS version of Ubuntu for at least 5 years a developer would be able to download a free version of Ubuntu LTS and this would be “binary compatible” with enterprises that are running the same Ubuntu LTS version. Ubuntu Pro adds an additional 5 years of support for certain packages ensuring that CVE patches are maintained. I don’t know if Canonical releases those CVE patches freely, so after 5 years you may loose the ability to have a “binary compatible” distribution to develop against for free. I tried to find documentation about Opensuse Leap which Suse says is derived from Suse Enterprise Linux. So I’m unsure if Opensuse Leap is “binary compatible” with Suse Enterprise Linux always, for a time period, or not at all.

One final point that I have heard made is that Red Hat has forgotten the benefit in open source of building on the shoulders of others. Red Hat makes a lot of money by taking packages with hours of work and labor like Apache, the Linux kernel, and the whole GNU stack adding their own special sauce to turn it into a enterprise grade distribution of Linux, but now they are not allowing others to stand upon their shoulders and build on their work. Perhaps a counter point to that one is that Red Hat does allow the open source community to build upon their work and their support of distributions like Centos Stream and Fedora. But if we would shift and look at the Ubuntu ecosystem. Canonical takes the work and labor of the Apache, the Linux Kernal, the GNU stack, and the Debian stack and releases an Ubuntu LTS often used by enterprises, and projects like Linux Mint, Pop!_OS, and Elementary stand on the shoulders of Canonical’s work.

In reality, Red Hat’s decision will not impact my use of FOSS. I’m still thankful to Red Hat for their support of Fedora, one of my favorite distributions, and one that I give my time to help out and improve. I’m thankful to the Red Hat employees who have done so much to further the Fedora distribution.

Perhaps the DL crew could consider that yes, their point that Red Hat “can” do this is true, but there might be more nuanced reasons whether Red Hat “should” do this and how it might impact their wonderful ecosystem that surrounds a great product built on and with FOSS. Also, @guss77 had a point worth considering that the “binary compatible” distributions don’t deserve the label “parasites” but were perhaps fulfilling a legitimate ecosystem need for application developers who want to build apps that will run on RHEL licensed systems upon release with as few bugs as possible.

Thanks DL hosts for bringing your thoughts to the table.

1 Like

I am curious where in their license agreement it states this. I am not disagreeing with you whether they do or not, I am just curious where this is so I can take a look myself. Do you happen to know where in their agreement it says this?

I’m looking at the “Red Hat Developer Subscriptions for Individuals” which you get when you try to apply for the free Dev license. I don’t have access to the customer license as IMO you need to first pay for a license, that is a minimum of $350 - which I don’t have.

Anyway, the “for individuals” agreement’s language is quite complicated - I believe in purpose, but here goes my reading of it. In the following, quotes surround verbatim copies of text from the agreement:

The agreement gives you permission for “running” the “Red Hat software” on some devices, as long as you are “an individual” and “not as a representative or on
behalf of an entity” (what ever that means, I believe it is a way to immediately disqualify people that carry labels such as from Alma Linux) - and there’s a short list of things that you are allowed to do with your “running” of “Red Hat Software” - so it’s definitely not allowed for any purpose.

Then the agreement says this: “if you use […] for any other purposes or beyond the parameters described in these Program
Terms, you are in violation of Red Hat’s Enterprise Agreement and are required to pay the Subscription fees that would apply to
such use, in addition to any and all other remedies available to Red Hat under applicable law. Examples of such violations include,
but are not limited to,
[…]
selling, distributing and/or rebranding the Red Hat Subscription Services (or any part thereof) contained in the Individual
Developer Subscriptions.”

The way I understand it (which I believe is the way it was meant to be understood, specifically re: the “not limited to” language) is that the agreement allows the subscribed user to run the software but nothing else, and they specifically call out “distribution”, in whole or in part, as an unallowed activity, as well as “rebranding” (which is what CentOS specifically said they did, and I believe Alma and Rocky do as well). The “remedies available under applicable law” reads to me as a lawsuit threat.

Whether recompiling the software, with or without modifications, for the purpose of “running” for an allowed use - is “beyond the parameters in these program terms” is left as an exercise to the reader.

1 Like

Rocky Linux can’t do better than this??

The arms race has started (when I say started, I mean - RedHat threw the first stone, but you need two sides to have a race).

The Rocky move seems to be trying to do an end run around an artificial limitation that was put in place specifically to prevent them from doing an end run - they treat it as a technical limitation: something caused by a static technical problem that they can’t fix so they do a workaround. Very engineery of them.

But there are humans on the other side - with conflicting views on what is “right”. RHEL must respond, either by giving up on preventing access to GPL sources (very very very unlikely) or retaliating - and as a result RHEL cloud users will suffer. I don’t know who will win, but I know who will lose - all of us.

I’m currently using UBI images to build a product, and I will now have to start working on an alternative solution. I’m not relying on source access to RHEL packages in the UBI images, but if RHEL changes the way UBI images are obtained and run - as a response to Rocky’s move - it will likely have an immediate negative effect on my business.

I’m not saying Rocky should drop this (and go silently into the night), but my livelihood depends on me and mine getting as far away from this kerfuffle as humanly possible.

Also see the Software Freedom Conservancy’s take on this issue: A Comprehensive Analysis of the GPL Issues With the Red Hat Enterprise Linux (RHEL) Business Model - Conservancy Blog - Software Freedom Conservancy

1 Like

Thank you, @guss77, for providing some well-reasoned objections to the presentation of the Red Hat content in this current episode. I was very disappointed by the coverage from the DL crew. the fact that there are a lot of clickbait-y provocative and alarmist headlines out there doesn’t justify a dismissive response to the real and legitimate concerns that people have about this move on Red Hat’s part. In fact, the DL crew didn’t address real concerns about the changes in source code distribution AT ALL. They only responded to the clickbait hype that they are complaining about. They “fed the trolls”, so to speak.

I think your two points are legitimate. Red Hat is effectively seeking a way to circumvent the GPL source distribution requirement. No matter how you dress it up or insist that they are following the GPL to the letter, they clearly aren’t adhering to the spirit of the license, when their own license basically negates the right of their customers to redistribute the source code. is worded in such a way that it seems to allow Red Hat to terminate a license based on a customer’s redistribution of source code. Red Hat should provide some clarity on that.

You’re also correct that there is a lot of value in having access to RHEL compatible systems in software and infrastructure lifecycles, and all the points you raise there are valid. I certainly don’t want to deal with licensing tracking and automation in a POC system that I’m going to be building and tearing down frequently.

Of course the Destination Linux folks can hold their own opinions, and share them with their audience. I just really found the presentation of the issues in this particular episode to be unprofessional. If their primary concern was countering what they perceived as misinformation, they did a poor job of it by being dismissive of legitimate concerns with this change.

1 Like

And I’d like to point out that Mike McGrath has been relatively explicit about going after Red Hat license holders who re-distribute the source code for the purpose of re-packaging the OS. Listen to him address this on a recent episode of the Ask Noah show. Specifically, start listening at 00:50:

2 Likes

Sure was good to hear from @kernellinux in this episode, which I think was awesome. There we have it all, in detail, and to my mind the interview makes things very clear and explicit and pretty-much as summarised by the DL team :slight_smile:

Update:
I found the explanation of CentOS Stream really inspiring and although I’d had a look at it when it first came out, I initally thought Alma would be quite good to try as an alternative to CentOS, because CentOS Stream clearly isn’t a replacement for CentOS. I’m not employed at the moment and need to get a contract first; thereafter when it comes to developing for open source, the KDE Project seems most attractive to me. As far as contributing for package maintenance for distros, I’d happily look at Debian, Fedora and CentOS Stream, though I’m also very interested in building Flatpaks because they can then be useful to more people. I hadn’t heard of ROSI mentioned in this interview and that sounds interesting too.

From my hearing of the interview with Noha Chellia, McGrath gripe seems to be (and I’m reading a bit between the lines here, though see time stamp 00:31:20) - that CIQ is selling support contracts (i.e. directly competes with Red Hat) on top of Rocky Linux - a project they are way too intimately involved in for this to be seen as anything other than “lets convince the community - and get Red Hat - to finance our costs for competing with Red Hat”. It is definitely a problem.

But to quash that, RH are basically destroying the free (both libre and gratis) RHEL rebuilders ecosystem - an ecosystem McGrath says he has no problem with (time stamp 00:31:45).

On top of that McGrath is also pleading innocence about source code availability - “oh, its all still there - on CentOS Stream” (not a verbatim quote, but the notion repeated several times during the interview) when that is clearly not the case, and I believe McGrath - as the high level executive that he is - knows that. As explained by a RedHattian (whose name currently escapes me) on Linux Unplugged 517 (https://linuxunplugged.com/517), CentOS Stream isn’t getting the exact source that lands in a RHEL release: even if you ignore synching issues, merge delays, release embargos and all that “fun” stuff (also discussed at length at LUP 517) and the intentional lack of “RedHat Release x.x” tags on the CentOS git repository, there is the issue that sometimes a change makes its way only to RHEL and not to CentOS Stream - in the example discussed in LUP 517, a patch is backported to RHEL 9.2, but for some (good technical) reason or another the commit to CentOS Stream is the rebase change that is intended for the next RHEL release, so the source for the package with the backport is never freely released. As the RedHattian says on LUP 517 (again, not verbatim) “whether CentOS Stream is the exact source of RHEL, the answer isn’t really either ‘true’ or ‘false’…”.

Regarding the CIQ situation - after educating myself a bit more on the issue, I’m willing to let the “parasites” comment to slide - if applied to QIC and QIC alone. Neither Alma Linux and its community, nor Scientific and all the others - whose only crime was to provide free builds to the community for no money - deserve that treatment.

1 Like

The Late Night Linux crew make a few points that are worth thinking about.

  1. Redhat revenues are multiple billions of dollars so this seems to be a move to take more of the money off the table, not because they are unprofitable or that they are in danger of being put out of business (at least hadn’t been – see 2, 3 and 4).
  2. This policy change may be targeting both Alma (Cloud Linux) and Rocky as they have a business selling support for their rhel clones.
  3. Point 2 was enabled only because they changed Centos to Centos Stream. That is, by killing Centos, they created Alma and Rocky.
  4. Somebody (I think Joe Ressington) suggest that this might spell the end for rhel in the long run because people may move towards more open free options like Ubuntu or Opensuse to learn system administration and then when they’re in a position to purchase support licenses, will be less likely to choose Redhat.

Bottom line is Redhat have been very profitable for many years and while Centos likely costed them some licenses, Centos (by far the most popular clone) was controlled by Redhat and did not offer paid support. Killing Centos produced Alma and Rocky who both also sell support licenses.

  1. The claim made by RedHat people - including in the McGrath interview linked above - is that in the past, the business model RedHat to support CentOS and similar rebuilders (Scientific Linux was around well before the CentOS fold) was that potential customers would start free by installing CentOS, develop expertise, and then want to buy commercial support for those installations, and an “upgrade” from CentOS to RHEL was basically installing 1 package. The claim is that this no longer (or ever) holds water and they don’t see that process as something that creates revenue. As such - and because there is some effort involved in this “debranding” support - they decide to save costs and drop the support.
  2. @zoof - I’m interested about the claim that Alma Linux has business support offerings: can you share more information? I have failed to locate any such offering from Alma, and also - initially - from Rocky. Eventually I had it explained that the main person behind Rocky is also the owner of a company called CIQ that offers Rocky Linux support services and claims to be the only provider offering “official” Rocky Linux support. This is obviously really bad optics - but I don’t know of any such problem with Alma. If there is - can you please explain?
  3. :person_shrugging:
  4. I don’t see it coming soon, or at all. RHEL enterprise customers make large moves - like replacing infrastructure OS - on the order of 10 years.
  1. “I’m interested about the claim that Alma Linux has business support offerings: can you share more information? I have failed to locate any such offering from Alma, and also - initially - from Rocky. Eventually I had it explained that the main person behind Rocky is also the owner of a company called CIQ that offers Rocky Linux support services and claims to be the only provider offering “official” Rocky Linux support. This is obviously really bad optics - but I don’t know of any such problem with Alma. If there is - can you please explain?”

    Alma Linux was created by the purveyors of Cloud Linux: AlmaLinux is born!!.

  2. :person_shrugging:
    You think Alma and Rocky would have happened in the absence of Centos → Centos Stream?

  3. “I don’t see it coming soon, or at all. RHEL enterprise customers make large moves - like replacing infrastructure OS - on the order of 10 years.”

    “long run”

You can paint it however you want but I think this move by Red Hat was really bad and I am not surprised after all that happened before even back then, when Red Hat acquired CentOS. Was it in 2014? Scientific Linux was the first victim. The latter project recommended to use CentOS to not duplicate the work. Now both are dead as we know them. This is not about clickbait or drama. It is about being able to criticize and have a real debate about this decision and its effects on the wider ecosystem.