scalability.org

Years ago, we helped a customer set up a Lustre 1.4.x system. This was … well … fun. And not in a good way. Right before the 1.6 transition, we had all sorts of problems. We skipped 1.6, and now we have set up a Lustre 1.8.2 system, and have several on quote now for various RFPs.

From our experience with the 1.8.2 system … I have to say, I have a sense that it is brittle. Yeah, you can call it “subtle and quick to anger”, or even praise some of the design features/elements.

It just has many moving parts, some work well (MDS), some … well … not so well (OST problem notifications). The failure surface is huge, and figuring out where you are on that surface has become effectively the morning cat-n-mouse game for us.

Speaking with other vendors, friends running these systems, I get the sense that people design/build Lustre systems defensively. That is, they know and appreciate that it will break, so they aim to limit the damage from this breakage. Control or limit the unknowns.

This could be something of a harsh assessment, but I just caught myself doing exactly this for a customer’s configuration. They requested Lustre, and we went back and designed defensively.

(more…)

Viewed 23166 times by 1610 viewers

… and not being able to.

Seems most of the folks at Sun/Oracle haven’t heard of Lustre. I had to explain it to them on several calls yesterday. They didn’t understand why someone would want to pay for support of a GPL licensed system …

er …

ah …

mebbe we found some real nice gotchas, and want to get Sun to work on them, and give us a hand in ameliorating them?

I know … too early into the acquisition to really know, but some of the responses have been interesting thus far.

If you know how to get a Lustre support contract out of Sun/Oracle, please, by all means, send me your telephone number. I’d like to shorten the time it takes to walk these trees until we accidentally find the right people.

(I shouldn’t complain, as I am so far behind on my own work in generating quotes … the irony is not lost on me … in our part it is due to a very healthy demand, and an ever expanding pipeline)

Viewed 18647 times by 1599 viewers

Sun/Oracle has decided to change strategy around Solaris.

But bad news is for the community as Open Solaris has an uncertain future. Oracle has made clear that not all features from Oracle Solaris will be added to Open Solaris. Oracle may not even release the code of most of the new features of Oracle Solaris thus keep an edge over UNIX competitors and community version. While price factor does not matter much, the code factor does matter a lot.

What does this do for Nexenta and others, with business dependencies upon OpenSolaris? We looked to OpenSolaris for a more up-to-date, less buggy Solaris. We are looking at this for one of our siCluster offerings … this might have to change now.

Makes sense from an Oracle perspective though.

Viewed 10647 times by 1483 viewers

SCO lost.

As I had said to some local colleagues who were (for reasons I could not grasp) swayed by SCO’s arguments, this would not end well for SCO. And it didn’t.

The game is effectively over. Its time to wind down SCO as an entity in an orderly manner, to distribute the remaining value to those that SCO owes money to.

Viewed 9071 times by 1379 viewers

ACLU seems to have taken down the BRCA gene patent from Myriad Genetics.

This could actually change a chunk of the drug development business model. I am not sure if this is a good thing (the business model change), though I also didn’t think that one could patent what is effectively naturally generated prior art.

Patents are about reducing theory to practice, and then providing a temporary monopoly on the use of that reduction to practice. This is the basis for small startups to discover a specific gene, what it controls, what impacts it, ask for venture money, and then go and develop a drug.

That monopoly has been, from first glance, torn away.

The impact of this could be huge. We’ll see.

Viewed 10456 times by 1441 viewers

Way back in the day, data centers used to be cold. Cold air came in, and usually in hot-aisle/cold-aisle configs, left through the back.

Power per rack was measured in a few thousand watts.

Cooling per rack could be mebbe one ton of AC. Up to two in the worst case.

Then stuff got denser. Somewhere along the line someone decided they could run their stuff at higher temperatures. This works fine for machines that are actually mostly open space (blades, sparsely populated server systems, …). It doesn’t work so well for densely populated server systems.

Inlet temps above 72F can be a problem for dense electronics. Poor airflow in a data center (e.g. no real positive pressure on inlet, no real negative (relative) pressure on outlet is a real problem.

Yet we’ve seen enough of our share of such data centers in the last 6 months that I am starting to question some of the designs I see. We might have to start actively asking customers, do you have the following conditions in your data center (and then list them), for optimal use case. If not, we’ll have to ask some defensive questions, such as, do you have inlet temperatures below 72F. Do you have positive front pressure, and negative back pressure.

(more…)

Viewed 9043 times by 1375 viewers

… yeah, the kidney stone saga continues. Had a basket extraction Wednesday, fine most of Thursday till evening, then Friday morning, they decided to remind me who was boss. Off the the ER I went, in terrible pain. Kidney stones are not life threatening, though there are times you wish death was less painful.

Well, now one of 3 has been removed (second extraction), other 2 will be blown up soon. Original one still a minor issue, hopefully no more work interruptions. Have a stent to relieve pressure. Trust me, that hurts. Have some nice meds to hopefully help me not worry so much about it either. Allowed to have coffee again (woot) after not having it since Tuesday …

Of course, during this time, trying to help customers out with issues, while drugged to the gills.

Viewed 6848 times by 1083 viewers

[update] trackbacks/pingbacks temporarily disabled. Waaay too much spam. Seriously.

From their updated user agreement:

License and warranty for your submissions to LinkedIn.
You own the information you provide LinkedIn under this Agreement, and may request its deletion at any time, unless you have shared information or content with others and they have not deleted it, or it was copied or stored by other users. Additionally, you grant LinkedIn a nonexclusive, irrevocable, worldwide, perpetual, unlimited, assignable, sublicenseable, fully paid up and royalty-free right to us to copy, prepare derivative works of, improve, distribute, publish, remove, retain, add, process, analyze, use and commercialize, in any way now known or in the future discovered, any information you provide, directly or indirectly to LinkedIn, including but not limited to any user generated content, ideas, concepts, techniques or data to the services, you submit to LinkedIn, without any further consent, notice and/or compensation to you or to any third parties. Any information you submit to us is at your own risk of loss as noted in Sections 2 and 3 of this Agreement.

They own you … or at least anything you say or can be linked to you saying.

Viewed 14057 times by 4450 viewers

Finishing a siCluster build for a customer. We need to see what we can do here. On the test track, and opening up the throttle wide.

(more…)

Viewed 10102 times by 1233 viewers

Well … more correctly, fixed the data model to be saner, so that the tools would be easier to develop and use. Still a few more things to do, and one (simple) presentation abstraction to set up.

The gist of it is that (apart from the automatically added nodes), adding nodes by hand should be easy. This also means by XML (not done yet, but I know how to do this), and web (basically XML or CGI like devices).

So I want to add a node into our database.


root@manager:/etc/cluster/bin# ./add_nodes.pl --index=4 --slot=4 --name=paul --location=rack5

Inserting node into cluster.db


And sure enough, its there …


root@manager:/etc/cluster# bin/ls_nodes.pl
george, eth3=10.100.1.1/255.255.0.0, ipmi=10.101.1.1/255.255.0.0, wifi=10.102.1.3/255.255.0.0[fast]
harry
paul


now lets attach a network interface to this node

(more…)

Viewed 16023 times by 1406 viewers