As tiburon progresses …

We are now booting: Redhat 6.1, Centos 6.0, Ubuntu 11.04, and others (including VMs!) via tiburon. Completely painless for compute and storage nodes. This is letting us get to the next phase: Application Specific Nodes (or “appliances on demand” in more common language). Basic idea is, spend zero … identically zero time on your expensive … Read moreAs tiburon progresses …

Big memory machines … part 2: This time with working riser cards

Yeah baby! top – 14:55:24 up 2:38, 1 user, load average: 0.13, 0.17, 0.17 Tasks: 697 total, 1 running, 696 sleeping, 0 stopped, 0 zombie Cpu(s): 0.1%us, 0.1%sy, 0.0%ni, 99.8%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 1009.840G total, 18.159G used, 991.680G free, 0.000k buffers Swap: 0.000k total, 0.000k used, 0.000k free, 148.590M cached PID USER PR … Read moreBig memory machines … part 2: This time with working riser cards

Update on the lawyer-bomb across our bow

Yeah, with all what we have going on, the last thing we need is a clueless company firing a lawyer-bomb across our bow. Remember that life isn’t fair, no time is ever a good time, and sh!t happens.
It seems that the whole purpose of their … er … communication … was to try to get us to buy the property rather than taking us to court for rent that is not due them. That is, effectively compel us to buy property we don’t want. Their lawyer indicated to our lawyer that this has been successful with another tenant.
I will not put in writing what I think of this.
I think they thought I would simply cave in the face of an aggressive lawyer.
Hell fraking no. I don’t scare easily.
There’s little risk of downside to fighting them … facts, statute, and case law are apparently on our side. We’ve suggested several times that they reconsider their merit-less case before it ever sees the interior of a courtroom, as its only going to cost them money, bad press, etc.

Read moreUpdate on the lawyer-bomb across our bow

[UPDATED with more info] regression in Gluster

Its looking like updates with older versions lying around didn’t make the new versions very happy. Actually made them very unhappy. More technical than this, it appears that a library search path found the older libs first, and they didn’t mesh well with the newer libs. This was with an rpm -Uvh upgrade at at that …
On an absolutely clean install, I cannot reproduce this problem. With an upgrade I seem to be able to reproduce.
So its not solved, but I have a work-around. The solution is to scrub the old libraries off (remove them from the search paths, etc).

Read more[UPDATED with more info] regression in Gluster

Moving web code base from Catalyst to Mojolicious

Its a long story. For those who don’t know, Catalyst is a Perl based web framework. So is Mojolicious. The person who started developing Catalyst years ago, left that group, and started Mojolicious later on.
I like many things about Catalyst. Like other MVC frameworks, it lets you divide up your logic between controllers (the heavy lifters), the model (aka the database), and the display logic. Prior to this, I wrote some rather ugly looking code which combined controllers and display logic. I usually punted on the model, as I rarely needed anything more complex than a simple database at best.
My main problem with Catalyst has always been its dependency radius (e.g. how much additional crap you have to install to run/maintain your code). I liked using it, but installing it was … well … a challenge. Especially due to the dependency tree including a number of hopelessly broken modules, specifically anything that matchs the regex /WWW::Mechanize/ has failed to build/test/install cleanly on every machine I’ve tried (all OSes I’ve tried, many different Perl’s). Some of Catalyst’s controllers have this module as a hard dependency … which I never liked … and thus caused updates and upgrades to fail.
But it got worse.

Read moreMoving web code base from Catalyst to Mojolicious

Monitoring tools

We have a collection of tools that we use for various monitoring. Some are the classical standards (iostat, vmstat, …), the somewhat more heavyweight (collectl, dstat), the simple (not in a bad way) graphical tools (munin, ganglia, …).
We’ve found tools like Zabbix do a good job of killing some machines, as there are often memory leaks in these tools.
What we’ve not found, anywhere, has been a good set of simple measurement tools that provide data in an easy manner that allow for easy inclusion into something akin to a dashboard. No, not talking Ganglia or Munin or others. They aren’t dashboards.

Read moreMonitoring tools