TCO: The sorta kinda but not-really argument, part 1 (the TCO study)

One of the more annoying parts of writing stuff for consumption online is that, every now and then, someone with an agenda, a very obvious agenda, will go off with weak arguments. One of those weak arguments that we saw yesterday was TCO. Not that TCO is a minor concern, it is a real, significant concern for management. How much something costs in the end is a sorely needed datum for any enterprise, company, entity, to make rational and realistic decisions.
That said, there are ways to use TCO, and there are ways to abuse and FUD with TCO. We saw my attempted fisker (I was the fiskee? I dunno, we need a new jargon here) hand wave, poorly, the TCO argument. We saw a weak argument, that companies now must hire new staff to support Linux.
Allow me to deconstruct and eviscerate this argument. It is wrong, it is illogical, it is weak. It is also the most common FUD we hear, so it is worth spending time to do this.

The major point of those raising this FUD is that Linux costs more to administer and run over the lifetime of the system. Assuming the hardware costs are the same, they start attacking the administrative costs. Their arguments fall into several broad categories.

  1. The TCO study said so
  2. Everyone knows how to administer Windows systems, and no one knows how to administer Linux systems
  3. You will have to hire more expensive staff to handle these systems

There may be others that I have missed, feel free to leave a comment, and we will cover them.

The TCO study said …

As we noted, there are several TCO studies out there. Numerous. Some of them (2004 time frame and before), such as the Yankee group indicate that Windows simply costs less to administer. The claims are varied, but one of the take home messages is

The 20% of Yankee Group survey respondents, who did possess the TCO data, indicated the most crucial issues and events that negatively and positively impact TCO and ROI, occur at the application layer and services portion of the software infrastructure and not at the core server operating system foundation layer.

Unfortunately for this study, its methodology appears to be questionable.

The research firm did not specify how it selected its respondents. Last year’s Yankee Group TCO study attracted criticism when it became clear that that the sample group was taken from a mailing list aimed at Windows system administrators.

More to the point, BusinessWeek did a better job of outing the problems and the bias of the major author, Laura DiDio

DiDio is one of the analysts who seems to have taken sides in the holy war over Windows and Linux. She published a ‘White Paper’ on the topic last year that was sponsored by Microsoft and posted prominently on the software giant’s Web site. Also, DiDio got caught up in the famous SCO case. When SCO first made its claims that IBM had misappropriated some of its code and handed it over to the Linux community, SCO showed samples to several analysts to prove its copyrights were being infringed. DiDio, a former journalist and not a programmer, was one of them. She reported that SCO’s claims seemed justified. She told me: “It appeared to be a direct cut and paste right down to the developers’ notes.” A couple of months ago, the judge in the case wrote that he had seen “an astonishing lack of evidence” backing up SCO’s claims.

Moreover, specific methodological problems are indicated. From this link we see

Yankee Group’s survey, it turns out, was based on the responses given by companies that had been selected from a mailing list devoted to Windows issues. The survey was funded and carried out by Sunbelt Software, a vendor of Windows utilities, which publicized[ed. spelling corrected] the survey through a mailing list called W2Knews, which bills itself as “The world’s first and largest e-zine designed for NT/2000 System Admins and Power Users”.
Sunbelt itself clearly identified the survey as being aimed at Windows system administrators. In the 16 February edition of W2Knews, which launched the survey, the company said it and Yankee Group were “surveying Windows Sites” to see how they were “responding to the Linux phenomenon and the TCO question”.

Ok. It is a self selecting study, on a windows focused list, of windows admins, responding to what they think of Linux vs Windows. This needs to be understood by anyone reading that. If it helps, the first 5 people wanting to read the study, if you send me your snail mail address, I will send you something that you ought to take with this study.
The BusinessWeek blog goes on.

I talked to marketing professors and other tech market researchers to ask what they thought about Web-based, self-selecting surveys. Their answer: they stink. You can’t post a survey on your Web site and let all comers decide if they want to fill it out. When I confronted DiDio with these conclusions on April 4, and told her I was skeptical of her results, she explained that the survey really wasn’t as she had described at all. It was done by a third-party survey outfit who solicited responses from a pool of qualified business computer users. “It really isn’t self-selecting,” she said.

And more to the point

One slide said ‘Half of Users Say Linux Deployment Is Cheaper than Windows.’ You might draw the conclusion that the other half say Windows is cheaper than Linux. But you’d be wrong. The bar chart on the slide showed that 34% of the respondents have not deployed a Linux server, so have no grounds for an opinion, and only 9% said their Linux deployments were more expensive than Windows deployments.

Remember, the TCO study take home message was that Linux was more expensive to deploy and manage. But their own data appears to contradict that. The article that all these quotes came from was a blog posting that did a fairly good job at eviscerating that TCO study. But those aren’t the only folks that took issue with it. So did TechWorld.
Their criticisms are also quite apropos, and biting

As such, the survey can only be said to be representative of system administrators that already use Windows, rather than sysadmins in general.

So lets do the same thing with say, I dunno, AIX and Solaris. Lets ask AIX admins whether or not a switch to Solaris will cost more. Such a survey would be as valid as the Yankee Group version, which is to say, it is of dubious value at best (it is a stretch to assign dubious value to it, I am being generous).
The survey of course ignored many other elements of the benefits analysis, specifically elements of the democratization of support (e.g. creating real competition for support services)

In the executive report, its author Laura Didio wrote that “a significant Linux deployment or total switch from Windows to Linux, would be three to four times more expensive and take three times as long to deploy as an upgrade from one version of Windows to newer Windows releases.”
However, Linux supporters say that such a claim knowingly gives only part of the picture in order to build the notion that Windows is cheaper than its open-source alternative. The survey failed to consider other important factors in switching operating systems, such as the freedom of choice that Linux makes available, since companies can easily change vendors and support contractors.

Thats right, there is no vendor lock in on the OS. We have switched from RedHat to SuSE in the past. We are examining a switch to or an addition of Canonical (Ubuntu).
This is some of what the Yankee group study missed. The bigger picture and how TCO fits into it. The horizontal scaling (e.g. replicating the Linux is free, in Windows, you have to pay for more licenses and CALs). Add in the bias of the principal analyst, the weak methodology that many others have done a much better job of discussing than I.
So when advocates and marketeers haul out this study, and they will, please, feel free to point them to the multitudes of critiques of it. Lets simply say that a fork has been stuck in this particular study. It is done, shown to be problematic in methodology and conclusions. It is not a study that someone seeking to make a valid point would knowingly or willingly rely upon to make their point. Paraphrasing Monty Python, it is an “ex-study“.
Next, the Forrester study. I did not see this one hauled out, but the TechWorld article linked above also decimates this.
There is a study published by Cybersource, that they updated after the Yankee group study came out. I looked for online criticism of the study but found none with a googling. If you find any, please bring it to my attention. In short this study created a model organization and examined the costs to support it. In the study, Cybersource notes that not all organizations are the same, and as theirs is a model of a particular size, may not accurately reflect all real situations.
Hey, thats brutal honesty. They are telling you that you need to be careful in drawing conclusions which you extrapolated to from this study without a detailed accounting of your own costs and design plugged in.
Also, Cybersource comes out and indicates, in the executive summary, where they are coming from. Go read the report, I won’t summarize.
One of the elements I really enjoyed about it was this:

What makes this particular TCO study important is that we explain, in sometimes droning detail, each and every step of our analysis. We also provide sources to every stipulated line item cost, for both Linux/open source and Microsoft Windows platforms. To our knowledge, no other TCO model sampling does this.
We hope to achieve two things by proffering this level of detail. Firstly, we want to dispel any notion that this study is in any way dubious. For each and every data point, you, the reader, are able to confirm our own research, results and conclusions. This provision marks the separation between hocuspocus reports and science. We don’t expect you to trust us. We expect you to either verify or refute our findings.
Secondly, by providing a detailed but generic model, we hope to provide you with the basis for your own TCO model. Simply take what we have provided and morph it into something which better befits your on organisation’s requirements. The core, the hard work if you will, has been done already and awaits your patronage. As an organisation considering the TCO differences between Linux and Windows, you should ultimately not trust anyone’s TCO study but your own. We provide you the basis for making this a reality.
Finally, if, for whatever reason, you feel that our model is lacking in any way or doesn’t provide a realworld implementation of a small/medium/enterprise computing infrastructure, we want you to tell us why. Contact Cybersource directly via <> and speak your mind.

That my gentle reader, takes guts. Putting out the model, the datum, and the methodology, not hiding it, and showing every step is the hallmark of something well thought out. Don’t trust. Verify.
But that whole line about “As an organisation considering the TCO differences between Linux and Windows, you should ultimately not trust anyone’s TCO study but your own” is critical. Advocates for windows solutions are going to haul out TCO after TCO that claims to show Linux will cost more. If they haven’t studied your organization, they really don’t know what it will cost you. If your organization and other elements are not well modeled by their assumptions (most are not), then you can’t point to them as being typical of what you would get. Just like standard benchmarks, you really need to take any TCO study that isn’t your own (well designed and modeled on your organization, measuring things of real value for you), with a few kilograms of salt. Anyone at all trying to convince you that their TCO report is valid is trying to sell you something.
In this Cybersource study that they tell you not to trust, but to verify on your own, we see significantly lower costs for Linux deployments and support costs as compared to Windows. Again, their model may not reflect yours. You can’t read this (or any other study) and claim it is universal and the definitive word. But do read this study. Really. They have a well thought out and well reasoned set of arguments and detailed discussion on other TCO studies, the good, the bad, and the ugly.
Cybersource notes that while you should not think of any study as conclusive, their review of Soreon’s study, also mentioned in a previous post, appears to provide similar methodology, independence, and comes to similar conclusions.

Published by Soreon/Research and Markets the report (titled ‘Saving Cash: A Comparison of Open Source and Proprietary Software’) presents a detailed TCO calculation of Linux and open source software and Microsoft software. It offers case histories, studies into licensing costs, wage and training costs, etc.
Out of the list presented, this study is perhaps the most clearly and openly independent. The firm which produced the report expressly establishes its methodology and offers its results.
Interestingly out of all the various TCO studies presented here to provide a contrast to ours as a separate data point, this independent research closely matches our own results, thus providing some corroborative value.

Some of the more amazing aspect of this study, the one you should not take as gospel, is that they modeled the costs for Windows and Linux consultants differently. Their words

As with most organisations that sustain an IT infrastructure, our example organisation will have the sporadic need to invoke industry sector experts to fulfil requirements which fall outside the knowledge boundaries and skillsets of the organisation’s core permanent staff. It is once again difficult to provide an accurate cost for these required services, but a realistic figure of $45,000 over the lifetime of this TCO model has been specified, and applied to the Microsoft solution.
We have however decided to apply a different cost structure to the Linux environment. In the first version of this document, we received feedback along the lines that Windows platforms would be easier and less costly to acquire external support for, as they have been in widespread use for a longer period of time. Our experience disputes this, but for the purposes of responding to these requests, we have weighted external consultancy fees for Linux differently than for Windows.
For both the Standard and Enterprise Linux solutions a figure of triple the allocation (to $135,000) of the Microsoft consultancy fee will be used. In the case of the Enterprise Linux solution this will be above and beyond the consultancy and support incorporated already with the Enterprise Linux products.

Read that last paragraph again. And again. Now remember Dan’s little FUD gem from yesterday. And remember that in their model, this study found, for the modeled organization, the costs to be lower for Linux over the lifetime.
As I indicated, this study is worth the read. Yes, Cybersource is an open source solutions company. They indicate as such throughout their document. Their study provides all the goes-in-to’s and goes-out-of’s (pronounced gah-Zin-tas, and gah-Zout-uhfs), that is the things that go into the model, the things that come out of the model, and the model. You don’t have to believe it, you can crunch it your self. None of this, the open source solutions company, the model, etc. invalidates their work. Whether or not their work is applicable to your organization is not as relevant as the fact that they walk you through how to do the TCO analysis. You can use their results (not advisable for any TCO study as they note), or crunch them yourself.
But this isn’t the only set of studies that conclude that Linux is less expensive.
Several studies that you will not hear the windows people talk about include the following.
The Robert Francis group study is one of the major ones. IBM talks about it on a web site. Do remember that IBM sells far more windows than linux. One might argue that this fact biases them in that direction. The RFG study has on it, title page, in big bolded letters so that there can be no misunderstanding:

IBM sponsored this study and analysis. This document exclusively reflects the analysis and opinions of the Robert Frances Group (RFG) author, who has final control of its content.

Yes, IBM paid them to study it. Draw your own conclusions as to whether or not is biased, given that IBM sells more windows than anything else.
IBM writes:

First, let’s look at the hard numbers. RFG examined pricing models and surveyed 20 companies, and puts the cost of an application server running on Linux (on Intel&tm; x86 hardware) at $40,149 over a three-year period, a 41-percent savings compared to the $67,559 cost of running the same software on Windows&tm; (also running on x86). The cost of running the same application server on a Solaris technology-based system (running on SPARC) is more than double that of Linux, totaling $86,478. Part of this is due to the fact that Linux systems support more applications and have higher utilization per server than Windows technology-based systems.

This is specific to J2EE application servers. Whether or not it carries over to other systems is unknown. But there are some issues for our next area, we will touch on in a moment.
IBM also mentioned a study by Pund-IT. This examined use cases of several organizations. This study is not as detailed as I would like. It does represent an overview of real world use and deployment cases though, and is useful in that context. To be fair, Microsoft will likely have several counter studies, that show how windows saved them time and money … er … over windows, and Linux of course. Again, useful datum may be found in these reports, as long as they were not scrubbed by marketing prior to publication. Every now and then, one of these comes out with some interesting real facts buried within, that somehow have escaped the de-information machinations. Your job, should you choose to accept it, is to explore this, and see if you can spot them. Use case can be informative, but you have to remember you may be reading it through a marketing RCG (read as: Rose Colored Glasses) filter.
Back to the IBM article. There are some interesting notes in here from the interview of the authors of these studies.

LER: Is moving to Linux still about the money?
Robinson: Yes, the money is still important. Anybody who can save money is a hero in a corporate environment today. Linux can help with that. But TCO is just one dimension of Linux. In our study, we found two big factors. First, that Linux is being used for more mission-critical workloads. And second, TCO has become just one factor of many benefits, which include Linux’s flexible licensing model, a wide range of supported hardware platforms, growing ISV support and fast administrator skill-set transfer from other UNIX?? platforms. That is a big mental shift that we’ve seen over the last couple of years.
King: We found that the main benefit that businesses saw in Linux was as a way to get out from underneath expensive operating-environment licenses. They also were looking for ways to get additional leverage out of existing hardware investments, as opposed to having to buy entirely new hardware and investing in entirely new operating system and application licenses. Those were the common “first-stage” benefits that companies experienced when they moved to Linux.

Paraphrasing Bill Clinton’s election mantra “Its the cost, stupid.”
TCO is an aspect. There are holistic elements that the TCO for a single server will not model. Such as the horizontal scaling. Such as hardware reuse (e.g. “older” fully depreciated hardware which is still supportable costs you very little to run once you pay the depreciation).
It’s the cost. TCO is a factor in it. Continuing on the TCO thread.

LER: What do you see as the greatest TCO opportunities for Linux in the near future?
Robinson: Linux will play well in consolidation strategies. With the Xen project, which is still in the early stages, virtualization is built into the Linux kernel. It’s very fast, very efficient and lets you get high levels of virtualization on a single server. Consolidation to reduce hardware count also improves reliability. With two separate servers, your reliability gets divided, not multiplied. If you expect a failure once every 100,000 hours, now you’re looking at every 60,000 hours. With four servers, it’s once every 12,000 hours. With too many servers, there are real risk factors for one of those nodes failing. Service-level agreements and things like that become easier to write, and you don’t have to worry about different failure rates.
King: There are going to be some interesting opportunities and choices around Linux this year, which are going to be driven in large part by Microsoft??. To get the full benefits of Windows Vista, which is supposed to be released this year, a lot of companies will have to consider hardware upgrades. At that point, the whole cost-benefit issue will come up for serious analysis.
There’s now recognition that IT can provide significant benefits to companies of almost any stripe. You would have to go a long way to find a business where IT doesn’t have some sort of influence on business processes. But at the same time people are trying to get as many benefits as they can out of the money they’ve already spent???and spend as little new money as they can???to attain those benefits. Linux plays well in those circumstances.

So there we have the TCO survey roundup. As I pointed out, as others have noted, TCO white papers ought to be taken with considerable skepticism. Specifically the Yankee Group report has been effectively eviscerated. If they won’t present their raw data, their model, their assumptions, their rationalization and reasoning, they cannot be trusted to be well representing what they actually saw. Such internal inconsistencies appear to plague the Yankee Group report as indicated in the BusinessWeek article. Other TCO studies of the same ilk are suspect at best. The TCO studies I noted above are in fact available for perusal, and do not appear to hide raw data or methodology. Whether or not you agree with their conclusions, they present their analysis in highly specific detail for you to evaluate them.
Regardless, if you insist on believing these studies without going through a detailed study on your own, note that it appears that the majority are showing slight to significant benefit in TCO to Linux. That is, if you go by the studies, windows advocates wouldn’t say

note he doesn???t try to make any TCO comparisons, perhaps because he???s read the independent studies that show Windows is cheaper in the long run

But of course, this assumes that such advocates are in fact aware of the controversy around the studies they hawk, and the thorough evisceration these studies have been dealt in the press.
Take home message: Try not to buy into such studies without thoroughly understanding them, and their limitations, and their applicability to your situation. Doing otherwise will leave you with egg on your face. Lets call this one done.