Some boot options considered harmful to performance

иконопис(BTW: still in London, then off to Stockholm, then home)
A customer just saw this with RHEL 6. Windows performance was higher than Linux performance on the same machine. The customer didn’t understand it, we guessed at first about it, and in the end our initial guess was wrong. But we caught what was wrong, with a WAG, and it troubled me. So I wrote this.
First clue as to the nature of the problem came from numastat. If you run numastat on a numa machine, you should get something like this:

landman@metal:~$ ssh jr5-lab numastat
                           node1           node0
numa_hit                11488947        17672711
numa_miss                      0               0
numa_foreign                   0               0
interleave_hit             27443           27648
local_node              11463950        17669861
other_node                 24997            2850

Two nodes, about equal access between them. Mostly hitting the right nodes.
Now a machine with a single node either has one CPU populated with RAM (and the other without), or is configured not to run NUMA at all.
Like this:

5 thoughts on “Some boot options considered harmful to performance”

  1. This is not exactly my area, but I hear stuff. 😉 What I’ve heard is that ACPI is a big ball of barely-documented crud that has never in its history been implemented properly, and that it works on Windows only because Intel and Microsoft engineers practically live in each others’ pants for months at a time – an effort Intel doesn’t extend to any other partner. Every other OS has to reverse-engineer half of it, and the cost of getting one thing wrong is system instability. When ACPI bugs turn out to be too intractable, the prudent choice sometimes is to disable it.
    But hey, those are just rumors.

  2. @Joe
    I’d be surprised if that was the RHEL default as their docs talk about how to enable it in grub if you are having problems and include a warning that some systems won’t boot properly without it.. But I don’t have any RHEL6 boxes to be able to confirm/refute that with I’m afraid. I’ll prod our local LUG in case anyone has.
    Whilst the kernel developers are not necessarily enamoured with ACPI (include the need to have an interpreter embedded in the kernel) it’s unfair to say that this is some private arrangement between Intel and Microsoft – Intel open sources their ACPI compiler and is an active contributer to the Linux kernels ACPI implementation (just try a “git log drivers/acpi”).
    Five of the top 10 committers to that kernel code are from Intel (all time they’re at #1, #2, #6, #8 and #10 and for the last 1,000 commits they’re at #1, #3, #5, #6 and #7), and there are also contributions from HP in there too (another co-developer of ACPI).

  3. @Joe
    A friend of mine at Oracle checked his OL6 boxes (both real and virtual) and none of them had acpi=off in grub. No response from people with real RHEL6 boxes though.

  4. (still in London, returning today … Stockholm was nice, wished I got to see more of it)
    Thanks. I am glad this is not a default. I am guessing it is a holdover from a previous install method. noapic is so … opteron … time. acpi is well, acpi. Kind of hard to have modern hardware without it.
    (snark noted and points added … well played!)
    Not saying acpi isn’t … an aweful mess … nor am I saying its the greatest thing since sliced bread. Its probably somewhere between these two.
    This said, its sort of hard to have modern hardware function in a complete/correct manner without acpi. That is, it is a necessary element. Not necessarily evil, but necessary. The evil portion is open to debate, and I thing from what I am hearing, the folks you’ve “heard things from” would definitely fall on the “lets stick a stake through its heart” side.
    This said, it was the only case I’d ever really seen of a substantial windows advantage over linux. This advantage was mitigated (and then some) when the correct booting options were used for Linux. Go figure.
    All this said, I think the end user is happier now than before. This is what matters.
    Keep your eyes open for strangeness on the boot command line. Simpler (and more verbose) is almost always a better choice.

Comments are closed.