# Tactics versus strategy for the HPC market

I have given the Microsoft entry into cluster computing a great deal of thought. I want to see if this is a force to be reckoned with, or something else. Will they matter in the long term?

A tactic is something you execute to further a long term goal. You may change tactics to achieve your goals. You may alter your tactical foci to adjust to market conditions. Individual tactics are not the important element, how they advance you towards your goals are. A tactic is not something you commit to.

A strategy is the big picture vision that you leverage in the process of achieving your goals. Strategies are often implemented by careful creation and execution of tactical efforts. You commit to a strategy, and create/change/drop tactics to implement the strategy.

Many people confuse these, refering to tactics as strategies and vice versa.

This is nice, but what does this have to do with HPC, supercomputing, or Microsoft?

Quite a bit I am afraid.

My initial thoughts were that Microsoft, a large multi tens-of-billions-of-dollars software house, with a few gamers thrown into the mix, had committed to high performance computing. That they saw this market for what it is, a vibrant, active, growing dynamic in which they would be able to offer value. My hope was that they would work to fit in and grow with the market, leveraging synergies, and exploiting the potential value they could bring. Rising tides, all boats, yadda yadda.

Then a few things happened. Words appeared in press. Conversations happened. Thoughts crystalized.

First, I had a talk with Kyril Faenov and Patrick O’Rourke of Microsoft. Both are nice people, and obviously Microsofties to the core. There is nothing wrong with that, and I admire any company that can engender such a strong bond among its employees. During our short conversation, Kyril implied that until Microsoft’s entry into the market, clustering was either hard, expensive, or impossible. I am not trying to misrepresent this, so if any of the people in that conversation wish to clarify/respond, please, be my guest. Their idea was that Microsoft’s entry would be the light-unto-clusters, showing us what could be done if Microsoft did it. That may be a somewhat flippant way to say it, but that is how it came across to me.

I’ll address this in a moment, but suffice it to say, that I found flaws in their premise.

Then I ran across a number of articles from various Microsoft PR folk, and others. Finally I ran into this. In this article, page 3, we see the plan.

Quoting “This is the year where we’re well-equipped to come back into the Linux strongholds and take some share. We have our Windows Compute Cluster Edition. We’ll get back into high-performance computing, which, at the end of the day, is something like 30 percent Linux servers.” This is in the section where Mr. Ballmer talks about how Linux isn’t growing, and how they are going to take share from something which isn’t growing.

One must wonder, if it isn’t growing, and hasn’t been growing, as he alleges, then where did that market share come from? Unix had been dying long before Linux became a serious player, Linux merely hastend Unixes demise. It couldn’t have been all Unix share that Linux took either. Since the server market was flat or slightly growing in this time, that market share must have come from somewhere. You can’t go from 0-X% without growing (for any X > 0).

This is beside the point. The point being HPC as a strategy in companies like Scalable Informatics, Basement Supercomputing, or Linux Networx, Panta, and many others, or HPC as a tactic, to ward off an encroaching competitor.

What we see is what Mr. Ballmer perceives to be a good tactic towards the strategy of how to deal with Linux as a threat. The tactic is to enter HPC and attack Linux market share. That is, HPC is not a core part of Microsoft’s business. It is at the periphery, at the fringes. They have left it before, as both RGB of the infamous Beowulf list indicated, as well as Mr. Ballmer himself alluded to in this article. HPC is a means to the end (their real goals and strategy). The end is to work on reducing Linux impact in the market, and the means to accomplish these goals are the tactics they will use. Microsoft’s HPC tactic.

HPC is not core to Microsoft. Not like it is at my company, or others with a similar HPC focus. We do HPC and anything that surrounds it. Little else. So when Microsoft, the worlds largest software company, says it is enteriing the market, we are hoping to see a strategy we can buy into. A grand vision. Something we can work with and add to.

Some of what we heard were things that we have been preaching for years (decades in some cases), using the same language we use. HPC must be easy to use. Must be transparent. And what our customers tell us again and again, it must be free from vendor lock-in. This is important. The first tuesday of every month, most of the corporate world deals with the aftereffects of a monocultural vendor lock-in. I haven’t met a single customer yet whom appreciates this.

Back to the point.

HPC is a tactic that Microsoft plans to use to get traction against Linux. An HPC strategy would include the realization that this market is going to be significantly larger over time, and that it is fueled by applications running on Linux clusters. A strategy would work within the existing market to grow it, enhance it, add value to it.

Remember that statement by Microsoft, that Linux clusters are too hard to use? If they are, then why are they growing as a market 60+% per year? Can’t be that hard. Or people wouldn’t buy them. What about their concern about integration? Again, this has not been an issue for our customers. They know how to use web pages, and the windows explorer to navigate. Therefore they can use the clusters we sell and support. What about their technologies such as integrated job schedulers? The market has a number of them out there, which had the features several years ago, that Microsoft talks about in their future.

That is, a strategy would not be one where you try to replace that which works well now, but to augment and grow that which works now. You want to hook in your scheduler? Here is a great DRMAA API to do it. You want to write “shell script code”? Fine, here is .Net on windows, and you know, you can do all the same stuff on the cluster with mono, so lets support that. Note that this would likely work well to help Microsoft defend itself against charges of monopolistic practices that it had been accused of in the past. You want to write web services APIs? Great here is the .Net/mono for it. Works across all platforms. Have nice front-ends on windows and linux, nice backends whereever. Lower the barriers.

Think of it this way. Java doesn’t run everywhere. There are no nice shiny new Java bits for Itanium2 boxen . You are stuck with older stuff if you have this. Use mono/.Net and the argument for Java pretty much vanishes overnight. But you have to support mono, and not merely grudgingly admit its existence until you bring your patents to bear. The nice thing is that with a blessed/supported mono, .Net now does run everywhere. Even architectures that .Net never anticipated. And the development and support are free. Lower the barriers.

These are things they could have done to show that they are serious about wanting to work with the exploding cluster market. Instead, they went in the raised barrier model. Proclaiming that all that the Linux clustering efforts were not reasonable, and theirs was reasonable. If Linux clustering were unreasonable for end users to implement, why then is it the engine pushing the growth in the HPC market? Seems to me that this argument fails in face of market data. No strawmen needed, their argument is decimated perfectly well by 3 years of IDC and others data.

They are a late-comer to this market. The market was in high gear before they arrived. Shows no signs of slowing down. Their product is not cost competitive with the current market leaders. You can load a 32 node cluster with Linux for $0USD software/tools acquisition cost, and get some of the highest quality scheduling, interoperability, web services, development, and other tools. Your high performance computing applications will work out of the box, as most are targetted at Linux and Linux clusters. You can be productive in many cases in under an hour from starting if you use default or preset configurations. This is what they are competiting with. Whether or not this was a good idea will be answered in the future. Since it appears to be a tactical path rather than a long term strategy, who knows how long Microsoft will be committed to this path. If it fails to pan out, they could drop it. Or not. I like telling people that systems designed to fail, often do. I do not the Microsoft HPC-as-a-tactic approach as being designed to succeed. It alienates the community of users whom have been and will continue to successfully drive HPC forward. I do hope that they revisit their thinking, and decide that HPC as a strategy is a good thing, and abandon the HPC as a tactic to attack Linux. The market would be much more interesting if they were going to work with the existing community. There is much that could be done together in this case. Microsoft may be banking on creative destruction, to replace the existing market with one of their own making. This would be good if they could demonstrate better/cheaper/faster. Where I sit now, I see more costly, and slower due to all the anti-viri and firewalls you must run (in corporate America anyway) on every windows machine, regardless of its function. Better? I disagree that this model is better. YMMV. Today you can do quite a bit with your Linux platform. From laptop to supercomputer. Easily. No patch-tuesday. It just works, it is easy to setup and manage. With regards to HPC strategies, this requires a vision of where you want to be. Mr. Gates did a good job of articulating what we and many others have. You need HPC to be integral and invisible. Has to just work. Briefly, my vision on HPC is that I would like to put on a data glove, enter a virtual world, adjust my atoms, and start my molecular dynamics simulation. Stop it, move some things, measure stuff. All in this virtual immersive world. A world where I can stop thinking about how to code up an FFT or xGEMM to get the optimal performance out of the silicon, and where I can start thinking about the science I can do with the tools. This is what I have been talking about since at least 1990, if not a little before. We are still a ways away. I believe that before I retire that I may get close to being able to do this. Today I can run the same code that I run on my supercomputers on my laptop. And on some PDAs. And cell phones. And game consoles. All of them run Linux. I suspect the system that will enable me to do that sort of immersive molecular dynamics will also likely be running Linux. Maybe something else, but it would need to be much better. Can’t get cheaper. Its not the platform that matters, its what you do with it. ### 7 thoughts on “Tactics versus strategy for the HPC market” 1. Hi Joe, I have some downtime in Dresden, so I’m catching up on my reading. Let me add some comments for you and your readers. The Windows CCS team is focused on the following strategy: In five years, we want every researcher and scientist to have supercomputing power from their desk. We want HPC technology to become a pervasive resource???something that???s as easy to locate and use as printers are today. The tactics include: getting the top application ISVs in each target vertical/industry to port to Windows x64; to work with OEMs and system builders on form factors and solutions for HPC; to engage academic institutions for feedback and ongoing research; leverage existing Windows infrastructure/tools so Windows CCS is simple to deploy, operate and integrate; interoperate with existing HPC technologies (i.e., MPICH, OpenMP, Platform LSF). The Windows CCS team believes the next revolution in HPC, whether scientific or commercial, will be driven by data. Broader access to personal supercomputing resources will lead to increased global collaboration and more meaningful data that can accelerate the time to insight. Talk soon. Patrick O’Rourke 2. Today, Linux provides desktop access to supercomputing power. Globus provides grid layers that enable you to use systems over a wide area. All of what you indicate is provided today. This was my point. HPC being pervasive is a great idea, one we have been pushing for quite a while (about 10 years). We agree on this, and we can build on this. This was a point I made about how I think Microsoft can work within the large and rapidly growing market. Unfortunately your management seems to make statements that are focused upon their pain points, that being Linux encroaching on their turf, so they will return the favor. The tactics I was refering to referenced the competitive landscape Microsoft finds itself in from Linux. Mr Ballmer’s words, and many of the other quotes I have read seem to indicate that Microsoft is viewing this as a way to attack a competitor, not to commit to a market. This has me puzzled to say the least. If Microsoft were committing to the market, wouldn’t they want to start working in the existing market? What I see is a market replacement tactic, in order to define a broader attack on Linux. What I don’t see is something that will be beneficial to customers in the sense of being demonstrably better (or even as good) as existing technology or price points. A few of us have some pretty grand visions about where we would like to see HPC go. The molecular dynamics bit I alluded to is one element. Others have similar basic ideas. I can see how as Linux develops and grows, it will address more and more of what I want to do until we can do the sort of supercomputing that we want to do (completely invisible to the end user unless they want to see it). I simply don’t see how Microsoft would get us there, as all I see are the Linux replacement stratagems. Every market revolution in HPC has been based upon a price performance win in the past, with the emerging victors providing incredible price performance advantages. Read some of my earlier posts on the evolution of the market. I have a pretty good idea of what this next revolution looks like (this is a whole other topic, one you may see other posts on this blog about). There are pain points as well. The pain points are going to determine the structure of the solution, and this plays well into what we are talking about. The hardest problem going forward is going to be data motion. I have been saying this for a while (several years). Some folks are architecting their high end supers to minimize data motion whenever possible. This is because data motion is hard, it is expensive. As data grows along its exponential curve it becomes exponentially harder and more costly to move it. This is as true on small clusters as it is on huge. Curiously we have a solution to some elements of this problem. And we are working organically on others. What I would like to see is Microsoft work with, not against, Linux clusters. Have .Net bless mono. Work with it. Focus upon lowering barriers. Work with DRMAA so we can easily (trivially) schedule across domains nodes. Use cygwin or a real POSIX environment so that we can easily integrate windows machines into existing clusters. Lower the barriers. The tactics I see all raise the barriers, inhibit interoperability, and force choices. Linux would be happy to play with Microsoft, Microsoft simply has to decide for itself that this is a good thing. It is. Work with us, not against us. Rising tide, all boats, yadda yadda yadda. 3. BTW: posted before coffee. A dangerous thing. The point I wanted to get across is that there is commonality of vision. There are hard problems, and it is better for the customers if we work together rather than at loggerheads. The Microsoft tactics I see going forward are only going to increase the problems not make them better, by creating solutions to already solved problems. My point is that it is better to see what end users need, and not just from a what-we-would-like-to-provide-them viewpoint. Kyril’s, Mr Ballmer’s and other statements about HPC and clustering being hard until now belie the facts, and are IMO not close to reality that we, IDC, and the cluster purchasing market see. HPC in general is hard, but this is a different problem. Cluster building is not hard per se, as long as you have a good design to work from. The design of course is dictated by what the user needs to do with it, not the demands of the OS. 4. Joe – I think it’s unrealistic to forget that Microsoft is a commercial enterprise, not an open source project. I’m not denigrating open-source projects at all, but I think you have to remember what their ultimate goal will always be: to sell more licenses. To raise your expectations beyond that is to set yourself up for disappointment. I also think that you are underestimating either the complexity of setting up and running a cluster or the size of the HPC market that Microsoft is trying to tap. For the most part, people who set up clusters right now are either Linux gurus, or they hire gurus (like yourself). They’re capable of recompiling a kernel when they need to, and of recompiling apps to run on their favorite flavor of the OS. They’re even capable of installing a new flavor of the OS in order to run a different app. (How many biology PhDs are perl experts because they need to be in order to use their cluster?) I think Microsoft wants to let a far greater audience tap into the power of HPC, and that means making things far easier to set up and use. They’re not trying just to make the existing HPC audience happy, but a far larger group of people. I think they will succeed in lowering some HPC barriers, and I hope to see them enable clustering on a far greater scale (that is, in more enterprises and SMBs) than ever before. All of that said, I still agree with much of what you say! I think that interoperability *helps* Microsoft. I would love to see them blessing (even fostering) Mono. I don’t think they’ve seen the light yet. But if and when they *do* see the light: it won’t be because they’ve got some higher calling has inspired them to interoperate “because it’s the right thing to do.” It’ll be because they see it as a way to secure their place and to sell more licenses. 5. Hi Dan: Tried the video last night and it didnt come through. A few points. No one is forgetting that Microsoft is a commercial enterprise. Least of all Microsoft. Of course they want to sell more licenses, but when you have a defacto monopoly on licenses as they now do, you need to figure out new and different things to do. Hence their xbox, and other efforts. That said, when you are competing in an environment that already exists, it behooves you to understand the market in the sense of being able to address market needs, supply value up and down the chain, and more importantly, not to price yourself out of the market. The market that Microsoft claims they are going after is somewhat off what they appear to be doing. Just got an interesting email that I cannot share w/o the authors permission (will ask), which shows Microsoft going after the heavy metal, not just the small/medium clusters as they had claimed. This will be released in a Microsoft press release soon, and it does a number on the arguments I have seen people make (including apparantly your boss on his blog) about Microsoft ignoring the larger machines. The “so-what” aspect comes in, but it puts in question the statements that Microsoft is ignoring the high end. They are not. They are focusing money/people/resources on it. I am sure they gave these licenses away for a song (or for the free press, it is FOIA-able, and I am sure someone will check). Not what mom-and-pop would pay for them. As for us underestimating the complexity of setting up and running a cluster … hmmm…. you must not know my companies business. We (and a few others) make this simple. We have gone from boxes on the floor at 10am to doing end user calculations on 144 cores slightly after noon. You overestimate and overstate the complexity. This may be as a result of digipede’s business, which appears to be directly competitive to Linux clusters. Please correct me if I am wrong about your business. Cluster setup/installation is not nearly as challenging as you and others have made it out to be. Most of the experts around know the statements we have seen eminating from Microsoft on the complexity of cluster setup to be Marketing FUD. We are of course not the only folks out there doing this, though we like to think we are pretty good at it. For non-experts, there are very popular tools that are *trivial* to set up baseline clusters with. You can set up hundreds of nodes in very little time from bare metal with Rocks and be productive right away. Similar for the other tools. Our tools are even simpler to use, and the folks using them so far love them. The biology PhD’s (I am a physics PhD so I can’t comment too directly on what they did back when I was in graduate school) are not Perl experts due to wanting/needing to use their cluster. I want to emphasize this, it is an important point. They are Perl experts because their science has rapidly gone from cataloging to information rich. Perl provides extremely powerful tools to process information, rapid application development, and a simple learning curve. Sure the Pythonic folks will contest most of this, as would Javans, but the fact remains that Perl is the lingua franca because it is so good at what it does. Whats nice about it is that unlike Java, it really is write once run everywhere. Now as to what it doesn’t do. Clusters. Perl is a programming language. You want to have it interoperate with a cluster, you need some sort of resource manager/job scheduler. SGE/Torque/LSF/Slurm/…. Combine those, and you have a very powerful program. I did this 6+ years ago when I created SGI GenomeCluster which was (as far as we knew then) the first commercial code to scale NCBI BLAST across clusters. What we did then was to hide the cluster and queuing and other bits from the user. They got a drop in replacement for blastall. Didn’t need to know a thing about the backend, and they didn’t care or have to care. Since then many folks have copied, modified, tweaked our basic idea. Some folks even went further and took the job scheduler out of the picture, writing their own. Every single person we showed this too was amazed. That was the ah-hah moment for me. I had meant for it to be a drop in replacement, but I hadn’t realized how important that was. All you had to do was to change the path to the binary. Use the same options. Someone else said in another thread that people want to think at higher levels of abstraction. I agree with this. Tools like this let you do it. I would say more, but there are proverbial cats in proverbial bags, and I want to keep it that way for the moment. As for Microsoft’s claim that they don’t care about the higher end, this will be shown (by Microsoft, in short order) to be untrue when they reveal their latest high end cluster user. Microsoft is targeting HPC, not just small/medium clusters. Remember that actions are worth infinitely more than marketing statements. As for making a far larger group of people happy, I think I disagree with this. They are targeting Linux using their HPC as a tactic approach to do this targeting. In order to do this, they are addressing the same market that consumes Linux clusters. This is great, though they made a number of critical errors: 1) if Linux clusters were too hard, then why are Linux clusters growing at 60% CAGR and currently represent an about$4B market?
2) if ISVs have been waiting for WCC, then why are the delivering and supporting their apps on Linux?
3) if customers have been waiting for WCC, then why have they been buying Linux clusters? And why has the Linux cluster market been exploding at very large growth rates?
What I am implying is that the marketing statements we and many others have heard are largely at odds with data from IDC, customers and so on.
Microsoft can either view this market as one to attack (which is what they appear to be doing), or one to grow with. The latter is simpler to do, and enables them to tap into a rapidly growing market. The former forces them to compete with a zero acquisition cost competitor that has already solved the technical problems that still are ahead of Microsoft, and provides a comfortable user interface that the people using the applications have been using for the past decade or more.
That is, their approach will raise barriers to adoption, increase end user costs as they will now have to purchase additional Microsoft licenses for something that was free and working before, and get something with no demonstrated added value.
The only way to make that successful is to layer products atop their offering (at additional expense, complexity). This is where digipede comes in from what I can tell. Nothing wrong with this, I like people to stand behind what they do for a living.
We build high performance computing systems. We design them to suit the problems, we deploy them, we support them. We help re-engineer them. The platform is a secondary issue as compared to the application, the use case, the need and the budgets. Linux clusters address all of these issues very nicely.
My points about WCC are that its story isn’t nearly as good, it costs more, comes with vulnerabilities and issues that you really don’t want on a high performance computing system. There were far better ways for Microsoft to address the market, lower barriers, and increase mindshare.

6. Joe – I appreciate your long and well-thought-out reply. Sorry I couldn’t respond earlier; this turned into one of those days.
Of course, I violently agree with some of it, and I disagree with other parts.
Where do I violently agree? Well, to start, I love the story of your “ah-hah” moment. To me, that’s the essence of where distributed computing needs to go: silently working behind the scenes, making things run faster. Much, much faster.
As far as the markets that Microsoft will be going after???I have no doubt that they will be pursuing the big iron as well as the small iron (if John thinks they’re *only* going to sell to small and medium businesses and departments, he’s wrong [not for the first time, I can tell you!]). But I do think that they will be pursuing the small iron more vigorously than anyone has ever pursued the small iron. They really want to make it *easy* to buy, install, and use a cluster. They want to make it feel exactly like setting up your Active Directory. Is that a good thing? I’m not saying it is; I just think that’s their goal.
Of course I’ve followed Scaleable Informatics a bit. And I have a question: if setting up a cluster were really very easy, would Scaleable Informatics have a business? I’m not trying to spread FUD here (I think companies like yours provide an extremely valuable service to their customers). I’m also not saying that it’s rocket surgery. However, if clusters and apps truly ran perfectly out of the box, customers wouldn’t need the help of consultants. I certainly don’t have a quibble with your tools or their ease-of-use (although I think your comment “Ours are even simpler to use” makes it clear that there, in fact, *is* room for improvement with the other tools you mentioned).
Believe it or not, there are people out there who never want to see this: http://www.scalableinformatics.com/public/ctop-1.jpg. There are people who never want to worry about whether the binaries they downloaded will run on SuSe, RedHat or Hoary Hedgehog Ubuntu. People are concerned with ease of use, and people like you can make a Linux cluster easy to use.
Now, to be perfectly forthright, if Microsoft had built an easy-to-use .NET scalability system into their OS or development tools, our customers wouldn’t need Digipede, either. We’re here because there’s an opportunity there. You asked me about Digipede’s business, and you were partly right.
But it is certainly *not* our business to tell people what platform to use. There are many good reasons to use Linux and of course many people using it effectively; however, many people prefer the MS platform (some for its ubiquity, some for its development tools, and some simply because someone in their organization made a decision a decade ago and they’re now stuck there). We’ve simply created a distributed computing toolset for the people who have already chosen a platform. I definitely don’t have anything against Linux, and I’ve never gone against Linux in a sale???we talk to people who made that decision long ago.
Your paragraph “Microsoft can either???” is pretty much right on. People who own Linux clusters for which they paid \$0 for OS license and from which they are getting good functionality aren’t going to suddenly decide they want to pop for a bunch of license fees and a new OS. Of course. If Microsoft is trying to wrest that market away, they’re going to have to work very hard (and you’re also right that they haven’t shown the slightest inkling how on earth they could do it). But for people who haven’t put in clusters because they don’t want a second OS in their data center (and, for many medium size businesses, this is a real possibility), MS has a legitimate shot.
The other place they’ve got a shot is by bringing distributed computing to people who don’t really have a way to get at it now: .NET developers. I’ll reiterate one thing: I really wish they’d foster the Mono project, help bring it up to date and keep it up to date.
Anyway, this has gotten long and rambling. I really do appreciate your response, and I reiterate that I have the utmost respect for the work that you do (I also follow the bioclusters and biobulletin boards, and I know how much you contribute to that community).

7. In the interests of preserving time, I’ll try to be short and answer your main points.
You say They really want to make it *easy* to buy, install, and use a cluster.
This presumes that it is currently not easy to buy, install, or use a cluster. This is what we (and a number of others) disagree with.
It is easy to buy a cluster. Our knowledge/understanding of a core set of codes/markets enables us to construct designs which are easy to purchase. More on that in a minute.
Ease of installation is in the eye of the beholder. The easiest cluster setup I have seen comes from the Rocks folks. This doesn’t it is not customizable. It is. It is just very easy to make it work (on the RHEL supported hardware, more on that in a minute).
Ease of use is also in the eye of the beholder. We have customers who are scary-smart with unix/linux. They teach us a thing or three. We have customers for whom “ls” or even “dir” is too complex. For them, a web page with the apps, run controls, and other bits is what they need. That and a tie-in to their disk, so they can use explorer to drag and drop to move their data. We have a nice solution there as well. Installed at a few customers, and they do like it. No, not really covered on our web page. One of the joys of being revenue backed means you get to make hard choices about what you have time to talk about in your “marketing” (read as web pages). Would love to spend more time and money on showing what we can do, but time/money are zero sum games at fixed points.
Ok, now lets talk about the designs a bit. I haven’t yet met a customer who likes everything about a particular cluster, they always want to make changes. Even if the overall design makes sense, often they have particular preferences or reasons to change. We do our best to help them understand the potential impact of the changes.
You ask whether or not our business revolves around clusters being hard? No, of course not. Anyone can pull down a copy of Rocks and install their own cluster. I want to emphasize this, installation is not a problem, subject to the supported hardware list. Before you focus upon that too much, notice the hardware restrictions and the HCL of WCC. Many users do precisely this. They pull down the distro, burn it to CDs, follow the prompts on screen, and then have a functional cluster in a short while. Might not be a very good cluster, or maybe it is super. This all depends upon how good a job of designing it for their problem they did.
There is a reason for that (hardware restrictions), and it has to do with what a software vendor is willing to support in terms of complexity and risk. This is where our value shines. We adapt the cluster to the problems, and we don’t try to force fit particular software/designs upon the project. We freely intermix products from multiple sources as they are the best products to use, and not because we have business relationships to maintain. We form the relationshios when we see the need and the quality of the product that meets the need.
Part of our value is that we know what we are doing, we know how to map problems into designs that are usable/functional, and we know how to problem solve and support our solutions. What really gets to our customers is that we know how to support our competitors solutions, and often do a far better job of supporting them then our competitors do on our competitors hardware. Call up someone from one of the bigger shops complaining about something related to a network problem, and they will tell you to go bug the vendor of the network system. Call us up, and you will get an answer. You are going to pay for it, but in the end you will note that it is worth it.
Another part of our value is raw firepower or horse power if you prefer. We tweak and tune our systems, and those of our competitors. Some of our competitors work hard to exclude us from competitive situations where benchmarking is involved. For good reason. A customer asked us a while ago how come our performance data on some set of benchmarks was 20-30% better than that from one of the three-letter companies on effectively identical hardware. Our answer is simply hire us and find out.
I must ask though, what is “rocket surgery” ?
As for easy to use, of course, there is always room for improvement. I personally think the person that invented modal dialog boxes should be hunted down and flogged until they admit the error of their ways. Then go after the person whom believes that every severe error in a long chain of severe errors should pop up their own windows …
Every system I have seen has problems in usability. None is perfect, or close to perfect.
As for ctop? That is a unix/linux thing. If you care you will use it. If you don’t, ctop is just a front end to a data collection/analysis back end. Present it any way you want. The data is there. On one of our clusters, the rough equivalent is presented in terms of number of CPU years consumed, the total available number of CPU years, and the percentage utilization. Managers eat that stuff up. Couldn’t give a rip about processes. They want the big picture. Then again, folks like me (command liners) like simple/informative tools.
As for binary compatibility, this is a very important point, and one worth exploring. Have a good hard long look here. This defines ABI, so that for two completely different distros, call them Larry, and Moe, if both are compliant with LSB 3.0, and your application is built against an LSB 3.0 distribution, you should have every expectation that your binary will just work without thinking about it. This is pretty close to absolutely true, apart from the applications which have hardwired distribution bits (specific paths/package names) somehow (stupidly) hardcoded into them. The app should work everywhere unless someone was getting lazy and not adhering to standards.
I regularly install RedHat packages on my SuSE machines without a problem. And vice versa. This isn’t an issue, though after speaking with Kyril, one might think it was. This is an issue with Unix. This is in large part why Linux killed Unix. One ABI/API. One code base.
You can make distro specific packages if you would like, but this generally makes more work for you if you need to support more than one distro. A few ISVs have not yet weaned themselves completely from the many unixes view and view each distro as its own platform. This is wrong, and they will eventually come around. The ones that have have managed to reduce their testing matrix, their build/support times, etc. You still need to run regression tests on every distro (some distros do some settings differently, and you want to catch that before your customers do), but these tend to be cosmetic in nature (install scripts, paths, permissions, acls, …)
As for the existence bits, I assume you (Digipede) are there because you fill a particular niche. Same with us.
We have dealt with customers running all manner of unix/linux/windows enviroments, sold and supported in all of them. The platform is only an issue if it is not capable of what the customer wants to do, or will increase costs, pain or other issues. As I have pointed out, the data strongly supports the notion that the level of pain is minimal for people considering these platforms.
There are folks who would prefer windows environments, and to that end WCC will be one way for them to address it. Not the only way, or even the best way. Some of us may have something to say and show about this, in a way that allows you to *safely* run your windows systems without firewalls or viri-checkers (this is not allowed in the corporate America that we deal with). This would be quite different than what WCC enables, as you have additional Norton or other licenses per machine, and the incredible slowdown associated with running those things.
As for seeing windows as a computing platform, we see it very little other than peoples desktops. What we are seeing more of are Linux desktops. Seen quite a few of these recently. Coupling this with some other bits of software we help them see how they can get integrated (single console handling both, without the need for KVM, or expensive Citrix like software). Linux laptops too.
People who decide to go for clusters have already made the decision to fit the solution to the problem, and not force the problem into a particular solution. This enables consideration of tools on their merits. The advent of WCC is not resulting in the sudden onset of ease of use, high performance, or integration and managment ability, certainly these features and functionality existed long prior to WCC. WCC is generating much sound and fury. We need to leave it to the market to decide if if signifies anything.