Ten years ago this blog was born

This was my first post. On 12-October-2005. I’ve written about many things over the past decade. 2000 plus posts, 200 per year, averages about 4 every 7 days or so. I’ve slowed down a bit in recent months, as work has grown more intense, but there are many thoughts I want to get down. To a large extent, my journey through HPC has been an interesting one, and only slightly captured in these posts. I’ve been a user of high performance and supercomputing systems since 1986. Yeah, really. I’ve written code for these systems on everything from card-punches and 300 Baud terminals for some Markov model calculations, along with the green lined paper for output. This code was to simulate a random load, a solar cell, and a rough guestimate of incoming sun for charging, in order to estimate charge state. Now I see examples of the systems that we were crudely modeling on IBM 3090 V180 machines in 1986, on the road sides. It helps that the solar cells have gotten better. I did some APL for optical modeling. It was fun, but … well … I’m not an optics person, so I lost interest in that. One of the harder ones was hand coded/optimized x86 assembly for a famous experiment (before it became famous for a prize awarded to the PI). This was very cool, and required that I write an experiment interface, and a timing controller compiler. With a terminate and stay resident program in DOS to handle the command and control. Start by turning off interrupts, because jitter sucks. Made debugging … a challenge. I optimized chemistry codes, rewrote critical sections of bioinformatics codes, and developed parallel shared nothing but drop in versions of other common bioinfo codes. There’s even a patent on that. I started working on accelerators in 2002, and called them APUs. AMD hired us to write white papers, and sure enough I used the word APU in there many times talking about accelerated computing. It seems to have caught on there. I’ve been highly critical of benchmarketing, poor benchmarking, poor analytics, bad load generation here. I’ve seen some people attack me here (and in other contexts) for pointing out failures of models, companies, etc. For example, nearly 10 years later, where is Windows in HPC? I’ve talked a little about this distributed workflow engine we created in the early 2000’s, that I open sourced, and simply ran out of time to work on. I put it up on git several years after I had stopped working on it. I really wanted to do more with it, but its hard to pay people when you can’t get revenue from projects you spend time on. It was very cool though, enabling you to run code effectively seamlessly on different systems by taking local resources into account when constructing jobs. I’ve talked about Tiburon, now SIOS, as our command and control plane for our systems. I got sick and tired of fighting the myriad of battles over how to boot/install/run systems, as new OS revisions often fixed a few things, and (badly) broken new things. We never wanted to build our own kernel, but once we realized that the backported patches we saw showing up were often the cause of significant instability or performance loss (or both). It was an easy decision. I can fix things I can see. I can’t fix things I can’t. Likewise, dealing with distro built toolchains for anything important became a nightmare. No-one knew how to correctly build an operational and up to date Perl. Python lagged (use Python3 already, dammit!). Other tools didn’t merely show their age, in some cases, they were completely rewritten and the old shipped project discarded long ago. It became more of a risk for us to use broken tools, than to craft our own toolchain. We’ve reported on records we’ve set, measurements we’ve made. Some have been fantastic. Some not so much. I’ve not written that much about our journey to raise capital. This has been a very time consuming and frustrating process. More than that, it incorporates some of the worst behavior I’ve ever seen … I used to think that some of our competitors were annoying … I can’t tell you how many times we’ve been asked to give away our IP, our company, our secrets, etc. by “partners”. Maybe I’ll write more about it someday. We’ll see. In this new decade of the blog, I wonder how long this and other “lengthy” social media will last. It used to be that blogs were (back when I started) a good way to get ideas and discussions out. But readers are fluid, flowing to the next medium. Many bloggers are now 140 character “micro”-bloggers (remember when they called it “micro”-blogging?). And they misuse the format by posting 25 part tweets. Which suggests they should just get a blog already. Which they had given up for twitter. The mind boggles … no really it does. And the irony of this is, that this blogging software will tweet a link to this post. This is at least humorous… performance art maybe? Every year in HPC has brought us closer to being able to do more with our systems, to reach farther, to think bigger. I’ve been honored to have been a small part of this. I am hopeful that I can remain part of this. And hopefully I’ll get back to writing more interesting things again soon. Stay tuned, good things are coming.