Red Hat, rebuilders, and paths forward for HPC

Red Hat has decided to change the way it handles its source distribution for its enterprise products.  This change impacts downstream rebuilders, such as Alma and Rocky Linux.  Not to mention Oracle.  This change has potentially very large implications down the line.  I'll get to this later.

Red Hat's business reasons for doing this are somewhat irrelevant, but the way that they communicated this to the wider community, wasn't.  It was an example in how to reduce your total available market (TAM) of users, by insulting a fair number of them.  Labeling those who use the technology and provide for a larger ecosystem, which drives commercial interest in maintaining that ecosystem, really wasn't wise.    Given the possibility of backlash to the decision and actions, this could have been handled more deftly.

To throw fuel atop this fire, a number of Red Hat employees wrote their own personal opinions (not reflecting Red Hat/IBM), where they labeled rebuilders as "freeloaders" or other such terms.  One article I read went as far as labeling CentOS project (pre-RH acquisition) as a "bad" open source project.  Specifically because it did not accept patches/contributions from the community.

This analysis is materially flawed.  The author went so far as to claim that the 100% bug for bug compatibility wasn't useful.  Calling it a "bad" project is simply a matter of opinion, and is quite subjective.  

With regard to the bug for bug compatibility, au contraire.  That was precisely the value of CentOS (pre-RH acquisition).  And Rocky and Alma post-RH acquisition and shutdown of CentOS as that bug for bug compatibility.

Rocky thinks they've found a work around.  Alma is (at last look) still thinking things through.

To be frank, this was an uneise move by IBM/Red Hat, simply because the extended RHEL-universe that CentOS had enabled (and Alma and Rocky continued), made it large enough to be a defacto place to work from.  With the removal of this, and the likely application of IBM lawyers to harrass and impede any work-arounds, it is far less likely that the HPC community will look at this being the case.

This makes more things in play than choice of distribution.

Let me explain this a bit.  In the past, Microsoft had made the assumptions that they could build clusters using Windows (various flavors), and therefore extract wonderful license fees for clustered systems.  This (economic) model was never going to work for any number of reasons (not even getting into the technical ones why this would have been a terrible idea).

Red Hat/IBM appears trying to move down that same direction.  Close out the "competition" who build off of the open source repositories you are required to supply.  Do this by placing this source behind a customer paywall.  With an explicit note indicating that access will be terminated if you redistribute ... er ... open source (read as freely redistributable) ... source code.   Yeah.  That's it.

Not a lawyer, so I couldn't comment in a legal way.  I could note that this, from a business perspective, doesn't appear to hold much water.  That is, until you account for the legions of lawyers at IBM's disposal.  It doesn't matter if they are wrong, they can outlast you in lawsuits.  Bleed you dry until you have no choice but to cry uncle.

Ethical?  Not likely.  Legal?  I doubt it, but hey, I'm not a lawyer, so I have no idea.  Likely to occur (e.g. IBM launching legal broadsides)?  Yes.

This is chess.  Not checkers.   This isn't 3 or higher dimensional chess.  You can see the piece ready to attack your high value target if you take the step you promise.  If you can afford years/decades of litigation with IBM to win on the principle, then hey, go for it.  Otherwise, here be dragons.

So now, the HPC community as a whole has to start considering alternatives.  Many years ago, when CentOS was first taken over, and Red Hat announced that one could not ship commercial systems with CentOS, I moved @scalableinfo 's systems to Debian.   This required a little work, though some code had been built specifically for RHEL/CentOS packaging (sigh) and had to be adapted.  Not a heavy lift really.

Ubuntu could be a reasonable destination, though Canonical seems to be making larger scale problems for everyone with snap/flatpak bits.  And I see them as possibly going down the same direction as Red Hat.

Another possibility is, and bear with me here, Microsoft.  Take Microsoft's linux build (I've not used it, so I don't know its basis).  Make it free for all.  Provide commercial support (should be a simple sale process for MSFT) where needed, and make it free otherwise for anyone who is building Linux systems.

That would be a wild ride, embrace, extend, ...

Then Amazon could do the same thing.  For the same reasons.  Offer Amazon Linux as a build with commerical support where desired, direct/simple tie-in to their cloud.

Basically, what I am saying, is similar to what I've been saying for a while.  The OS one chooses these days is less important, a detail if you will, of the job you are going to run.  Which may be container based, or ...

All this said, as noted, the HPC community has some choices ahead.  These choices are now completely reasonable given the landscape change imposed by RH.  And given that RH is usually based upon an ancient userspace, and a (horribly  backported mixed) kernel, I'd actually say that they just foot-gunned, lost most of their toes, and are setting up their next shot.

I doubt Microsoft, Amazon, etc. will allow this opportunity to go to waste.  I know I wouldn't if I were working for any of them.

This is important, given the fact that AI/ML and LLMs are all built atop HPC technology.  And that is a growing, if not an exploding, market.  I could see NVidia step into the void to fill the need.  Or AMD.   Or a consortium.

The basic idea is that by trying to take all their marbles into their arms, Red Hat opened the doors to customers looking at a wider set of technologies not under control of a single vendor.  This isn't a Red Hat vs Debian vs whomever issue.  This is a "who controls basic infrastructure for a fast growing market" going forward issue.  Red Hat has exposed its underbelly, and has placed its entire market at risk by this move.

We do live in interesting times.