back from the UK, and a good reason to drink Guinness

I enjoyed my trip. Well, not the part of being away from my family, but there is much to see/experience in London. A curious difference between London and the UK in general and the US is the apparent lack of public restrooms (or WC’s if I have the right localization). Especially in a crowded space like Covent Garden.
The customers in the UK (spent time with two at their sites, on the phone with ~5 across the world, and working with ~4 via email) have good problems (not as in blocking problems, but emergent problems that occur in a variety of use cases). These are the sorts of things our services side handles well.
Its interesting to see how the configuration management space is changing, and we got to see the often interesting and unexpected on our platforms with newer Red Hat loads. I keep hearing how RHEL 6.2 has improved IO performance, but we just aren’t seeing it. Worse, actually, and there are some very significant stability issues on our gear. A quick bump to an updated 3.2.x kernel solved the latter. The former comes from a variety of sources.
Specifically, what we’ve noticed (initially suggested by a customer who caught first wind of this, by looking at performance and average balance) is that there are now some significant issues with load balance and IO write performance on multiple RAIDs using the deadline scheduler in the stock kernels (as well as the 3.2.x kernels). I started writing a parser for fio output to specifically measure this imbalance. Once we get more data (in the lab), we’ll do some analysis and post results.
Another thing I’ve noticed is the preponderance of puppet. Our finishing scripts perform functions that can be driven by puppet, so we are looking at doing some level of integration with this, in order to enable customers to reload our units if they use Puppet. Since we are also perl afficiandos, we will probably start using/supporting Rex for our storage clusters when running statefully.
I can’t say again how much I’ve enjoyed the trip. The people (the ones I worked with and spoke with) were/are wonderful. The environment was pleasant. The historical things to see (science, computing, culture …) were simply too numerous to visit in a very short window, never mind many trips.
And then there was the beer.

Read moreback from the UK, and a good reason to drink Guinness

The blame game

This isn’t what you might think from the title. Its an observation. I hope I don’t misstate what I intend to say, so feel free to chime in if you don’t agree with the wording. When you have a situation where a customer has a set of vendors, and a problem that needs resolution, the … Read more The blame game

Busy … as usual

I’ve been readying a UPS ripping post (not the function, but the company). May backburner that for now. But been really REALLY busy. This is very good, in all aspects. Have lots of opportunities, many quotes out, some telegraphed orders (we know they are coming, we’ve been told, working its way through the systems), … … Read more Busy … as usual