scalability.org

Typecasting and the “trust us” factor

Finding myself on the other side of the table in the consumer-vendor relationship has resulted in some eye opening experiences. These are things I look back on, and realize that I strenuously avoided doing during my Scalable days. But I see everyone doing it now, as they try to sell me stuff, or convince me to use things.

One of the eye opening things is a bit of typecasting of sorts. This typecasting is, think of it this way, assigning roles and capabilities to people based upon some external factor or attribute of the person. More in a moment.

The other eye opening thing is the degree of “trust us, we know what we are doing.” Not “hey, we’ll give you a deep dive into how all of this works” as the basis of the why-we-think-it-is-better. Sometimes this worked against us, as our now well educated customers would go off armed with more knowledge and use that to beat our competitors over the head, usually to get them to promise to deliver what we delivered. Rarely if ever did it really work, but it gave the purchasing folks a reason to hang their hat on as to why they should buy an inferior more costly solution, which didn’t solve the core problem.

But I digress.

The “trust us, we know what we are doing” goes hand in hand with they typecasting. You see, it seems that vendors like counting the numbers of Ph.D. types they have, pointing out that Ph.D.’s are smart, and hard to keep happy unless they are working on hard problems. And because they are smart, we should simply trust that what they are doing is right.

Now my business card doesn’t indicate my Ph.D. in computational physics.

But I purposefully sit there stone faced when they tell me these things, and ask relevant questions. How have they done X, what did they do about Y.

Some of the time, I am told, don’t worry my pretty little head (well, not literally, but you know what I mean).

Some of the time, I am reassured that their Ph.D. is very smart, and wouldn’t get this wrong.

Hmmm. Maybe I’m doing the business thing wrong. Mebbe I should style my name as Dr. Joe on my business cards. I earned it after all. Says so on the degree.

I dunno.

But what I find with these folks are that, despite their uncountably infinite number of “Really Smart Ph.D.s(&tm;)”, sometimes … they make mistakes. And when they do, watching product managers, sales execs, and application engineers sputter while trying to answer basic questions is not helping my confidence in their abilities.

Know what would? Sending those “Really Smart Ph.D.s(&tm;)” over so I can talk with them, maybe mind meld or something, so as not to play effectively telephone[1] between people who really understand the problems they are trying to solve, and the people who claim to have solved them.

Seriously, they keep all their Ph.D.’s at the office. Probably lured them in with tasty snacks or something. Then start feeding them these hard problems they think they have.

When in reality, having a Ph.D. doesn’t grant you immunity from making mistakes, doesn’t really mean you are super smart … actually could be the opposite, depending upon the program you went through … could be better correlated with severe masochistic tendencies.

Its that typecast of really smart people with some advanced degree, in a back room somewhere, that I find … well … interesting. And based upon that, I see quite a bit of “trust us, we have these people, they are eating their noon snack, and are totally dedicated to your problems”.

Erm … ok.

The only time I ever let myself be referred to as Dr. in the past was for press releases at Scalable, presentations I gave in front of large audiences, or when I taught classes. Otherwise I’m just an ordinary Joe.

If it makes conversations easier to have then, mebbe, I’ll start styling myself with the title I’ve earned two decades ago. Though I suspect I’d be locked in a back room somewhere, given snacks and fed hard problems …

[1] See https://www.linkedin.com/pulse/20140917154833-7202644-effective-communication-stop-playing-the-telephone-game/

How to handle curious conversations … part 1 of a few billion

So …

Suppose someone comes up to you and makes a claim. This claim isn’t backed by facts, merely by unicorns, rainbows, and their own biases. Yeah, this kind of relates to the previous post.

They argue based upon the claim. Stake out their ground. Insist that “none shall pass” in a black knight, Monty Python esq manner.

But they are wrong. Simply, factually wrong. Regardless of their biases, you and many others have been demonstrating the very thing that is claimed to be impossible, to customers for years. And helping them achieve it. It’s used, in production, working very well for years …

There are many approaches to handling this. One, seemingly favored by some these days, is to go in with rhetorical guns ablaze. Others, prefer to try to snipe specific points.

My recommendation … after nearly 20 years of parenting is this: Pick your battles. Is this one so very important? Is everyone on the internet wrong, and must they be corrected? Or can you simply not engage in an argument, with someone who likes to argue for the sake of argument?

That is, sometimes, I simply refuse to engage if there is no real upside. I can work with people whom I don’t agree with, on projects I don’t agree with, as I am focused on the mission. I don’t want to lose the war for the price of winning a battle.

One if the issues I think I see, not in this specific case, but in others on the interwebs, is a tendency towards infantilism. It manifests in a “it’s my way or the highway, and I’m gonna hold my breath until I get my way, or some attention.” How … just how … is this helpful?

Jeff Bezos has a philosophy that I like. You can disagree, but then you commit to the path you collectively agree upon. Even if there remains disagreement. You air your concerns, respectfully, and then move past that.

Why aren’t we (collectively) doing more of this?

On technology zealotry

I’ve encountered this in my career, at many places. Sadly, early in my career, I participated in some of this.

You are a zealot for a particular form of tech if you can see it do no wrong, and decry reports of issues or problems as “attacks”. You are a zealot against a particular form of tech if you cannot see it as a potentially useful and valuable portion of a solution stack, and (often gleefully) amplify reports of issues or problems.

Worse is the denial aspect though. Handwaving away real issues as somebody-elses-problem when you are pro-specific tech, and assigning fault for often poorly thought out or implemented bits when you are anti-specific tech.

This … really … gets old.

Tech is a tool. It either works well for a use case, or it doesn’t. You always want to be on the lookout for new tech that might help, without excluding it based upon your own biases.

I’ve used many OSes over 30+ years. From DOS, Windows, OS2, many unices. I have preferences, but I am not wedded to any one in particular.

I do object when people cast misplaced aspersions on tech, or misrepresent things to try to sway decision making. This reveals more of their own issues than the tech’s issues.

I do object that when shown clear, and unambiguous evidence of core issues, that some people refuse to make the mental leap to accept that their favorite tech may be a source of problems.

This wastes everyone’s time.

Sadly this results in bad decisions which result in years of technical debt. This technical debt may have huge momentum, and not be something you can unwind.

Just because of someone’s bias.

Yeah, it gets real damn old.

Interesting post on mixed integer programming for diets … that has some hilarious output

I am a fan of the Julia language. Tremendously powerful analytical environment, compiled, high performance, easy to understand and use, strongly typed, … there’s a long list of reasons why I like it. If you are doing analytics, modeling, computation in other languages, it is definitely worth a look. Think of it as python, compiled, with multiple dispatch and strong typing … and no indent-as-structure problem.

My Julia fanboi-ism aside, there was an interesting blog post about using JuMP, a linear programming environment for Julia. I’ve not done much work with linear programming, or mixed integer programming, so I read it hoping to learn about it.

As with articles like this, I learned new things about Julia in the process, specifically about a number of packages I’d not known about before, and capabilities that they add.

While I am always grateful for another opportunity to learn something, I found the presentation to be quite … well … humorous.

[…]
2353 grams Tea, regular, white, brewed from leaf or teabags, with cows milk not further defined
32011 grams Water, rainwater or tank water
148 grams Milk, almond, fluid
2848 grams Milk & water, skim cow’s milk & tap water
[…]

2 Problems:

Firstly, it is suggesting dringking 32L of water in a day. Why is that? This is because water has basically no nutritional impact to our constraints, so it can take almost any value. To solve this, a simple trick is to add a small factor to out objective, saying ???All other things being equal, a diet with less total mass, is better???. This can be done by adding a term + 0.01sum(diet). The 0.001 is small enough that it shouldn???t impact the true objective much.

Secondly, it recommends drinking over 2 kg of tea, and 2 kg of watered milk. That is pretty boring, so lets add a constraint that one shouldn’t ever consume >500gram of the same item.

Yes, the solver suggests drinking 32kg of water, 2kg of tea, and almost 3kg of watered down milk.

For those who don’t quite remember, 1kg is about 2.2lbs, so 32kg is north of 65 lbs of water. Per day.

But wait … there’s more!

The author attempts some sanity constraints in addition to the nutritional guidelines, like reducing moisture. And then they get this:

500 grams Salt substitute, potassium chloride
1 grams Salt, flavoured
500 grams Cream of tartar, dry powder
2 grams Caffeine
13 grams Calcium
8 grams Fibre
500 grams Folic acid
9 grams Iodine
500 grams Thiamin

Oh dear …

It keeps going from there. The author did include a disclaimer about trying any of the diets … basically … don’t. Check with your Dr first for any diet. Especially if you think you need to eat 1/2 kg of KCl, or cream of tartar …

Ok …

While it was funny to a degree, I did enjoy the tutorial on JuMP, and the capabilities of Julia.

Distribution package dependency radii, or why distros may be doomed

I am a sucker for a good editor. I like atom. Don’t yell at me. Its pretty good for my use cases. It has lots of nice extensions I can and have used.

Atom is not without its dependencies though. Installing it, which should be relatively simple, turns out to be … well … interesting.

[root@centos7build nyble]# rpm -ivh ~/atom.x86_64.rpm 
error: Failed dependencies:
	libXss.so.1()(64bit) is needed by atom-1.26.0-0.1.x86_64

In searching the interwebs for what Xss is, I happened across this little tidbit

Experiencing this on CentOS 7 as well. Running yum install libXScrnSaver works there as a workaround.

While you might think, hey great, he found something that worked, let me briefly describe this machine to you, so we can decide if it is really “great”.

This is a HVM, running KVM, with no display. It is a build machine I set up for building my NyBLE stack. Its running locally on one of my 10 year old boxen. I wanted to put atom on there, as it is on my gigabit lan, and I don’t mind remote X here at home. The editor performs nicely from my laptop.

That is, in order to use a text editor, I had to install a screensaver on a headless VM.

Ok, this isn’t quite the worst of it. While I was researching what the minimal installation of CentOS I could get away with and have something fully functional for my purposes, I started looking into the package groupings. Take the “Minimal Install” grouping, which I would assume, would install the bare minimum I need to get an operational system.

[root@centos7build ~]# yum group info "Minimal Install"
...
Environment Group: Minimal Install
 Environment-Id: minimal
 Description: Basic functionality.
 Mandatory Groups:
   +core
 Optional Groups:
   +debugging

Ok, it is build upon the core group. Lets look at that.

[root@centos7build ~]# yum group info "core"
Loaded plugins: fastestmirror
...
Group: Core
 Group-Id: core
 Description: Smallest possible installation.
 Mandatory Packages:
    audit
    basesystem
    bash
   +btrfs-progs
    coreutils
    cronie
    curl
    dhclient
    e2fsprogs
    filesystem
   +firewalld
    glibc
    hostname
    initscripts
    iproute
    iprutils
    iptables
    iputils
    irqbalance
    kbd
    kexec-tools
    less
    man-db
    ncurses
    openssh-clients
    openssh-server
    parted
    passwd
    plymouth
    policycoreutils
    procps-ng
    rootfiles
    rpm
    rsyslog
    selinux-policy-targeted
    setup
    shadow-utils
    sudo
    systemd
    tar
   +tuned
    util-linux
    vim-minimal
    xfsprogs
    yum
 Default Packages:
   +NetworkManager
   +NetworkManager-team
   +NetworkManager-tui
   +NetworkManager-wifi
   +aic94xx-firmware
   +alsa-firmware
    biosdevname
    dracut-config-rescue
   +ivtv-firmware
   +iwl100-firmware
   +iwl1000-firmware
   +iwl105-firmware
   +iwl135-firmware
   +iwl2000-firmware
   +iwl2030-firmware
   +iwl3160-firmware
   +iwl3945-firmware
   +iwl4965-firmware
   +iwl5000-firmware
   +iwl5150-firmware
   +iwl6000-firmware
   +iwl6000g2a-firmware
   +iwl6000g2b-firmware
   +iwl6050-firmware
   +iwl7260-firmware
   +iwl7265-firmware
    kernel-tools
    libsysfs
    linux-firmware
    microcode_ctl
   +postfix
 Optional Packages:
   dracut-config-generic
   dracut-fips
   dracut-fips-aesni
   dracut-network
   initial-setup
   openssh-keycat
   rdma-core
   selinux-policy-mls
   tboot

Look at all that nice firmware I don’t need/want. As part of core. Which is part of “Minimal Install”.

And NetworkManager. And postfix.

The problem is, when you install other packages, they will pull often large additional swaths of dependencies in to provide a library or a function which isn’t really used or needed.

Like the libXss with atom.

What I want, as a user, is to be able to build/install a minimal sized effectively appliance OS. I don’t want/need massive dependency radii. In fact, such a thing is an anti-pattern. It is the opposite of what I need.

Assume I am building minimal sized installations as appliances I can deploy quickly and easily. Do I really need LaTeX in, because of an obscure tertiary or worse dependency graph?

No.

And, to make matters worse, if you try to excise the unneeded packages, you invariably risk the integrity of the package set you are attempting to install. As that dependency graph is unforgiving.

If I were a conspiracy theorist (I am not), I might follow a flight of fancy over distro lockin with extended dependency radii.

I am, or like to think of myself as, pragmatic, which means I try not to let things like this get in the way of doing what I need to get done. And please understand, these dependency radii are inhibitors, barriers to be overcome in many cases.

They aren’t helpful.

I don’t see distros adding much value beyond bug databases and some packaging/tooling. When the latter gets in the way of the former, I’ll do what I can to contain the damage.

I know some people are as wedded to their distros as others are to OSes. Distros, OSes, etc. are details of an instance. Not the purpose of the instance. They are tools to be used, changed when they don’t work for something that does work. No one tool is perfect, no one distro, OS, kernel, toolchain is beyond question.

This is one of the reasons I question the longevity of distros that attempt to enforce their world view on wide package radii. There is a long tail of users for everything, and the ardent defenders of a particular way of thinking will defend it to the last developer. Even as the world moves on and leaves them behind.

Lets encourage the distro people to rethink what it is they deliver. So minimal is really minimal. There is far more value in that, than in trying to tie everything through package management and package dependency graphs that thwart peoples attempts to reduce footprint and increase efficiency.

NyBLE

So there I am updating my new repository to enable using zfs in the ramboot images. This is a simplification and continuation of the previous work I did a few years ago, with some massive code cleanups. And sadly, no documentation yet. Will fix soon, but for now, I am trying to hit the major functionality points.

NyBLE is a linux environment for hypervisor hosts. It builds on the old open source SIOS work, and extends it in significant ways. Basically, we want to be able to PXE boot (or locally boot if you prefer, but managing OS or boot drives is something you never want to consider at scale) a full working OS into a ramdisk.

I have this working nicely for Debian 9, 8 and others, CentOS 7 (and RHEL 7 by extension). Eventually I’ll add others.

The purpose of NyBLE is to build what we want to use, not what the distro or image builder wants us to use. It is not a linux-from-scratch. It starts with a base distribution, so as to make questions about “but its X and we like Y, so why are you forcing us to change?” largely irrelevant.

OSes are, generally, a detail of an implementation. Distros are packagings of OSes in particular ways, that in theory, should impart some value to the users and consumers.

In the past, this may have been quite true. Currently, the only real values distros provide are centralized bug databases, security patches, and approximately working configurations.

Distros carry significant baggage. Ancient toolchains, often EOLed for years are being shipped with “modern” distros. Just remember they are more “stable”. Or something.

Distros often have unfortunate dependency radii. You can’t simply install package X, as it also depends on some functionality on Y and Z, which themselves depend upon … . Which means, building small footprint installations with all the functionality you need, say, ideal for memory booting, becomes fraught with peril of unmet dependencies.

While this is all … well … a fun challenge, think of it as fitting together puzzle pieces, where you only approximately know the shape of the pieces and the picture they display. And of course, each attempt iteration at fitting things together, could take minutes to hours …

… now add to this a temporal dimension. Suppose you want to compile support for a subsystem in the kernel. This support requires that the kernel-devel tree exist for a number of reasons. And suppose the kernel-devel tree is very large. Which makes PXE booting impossible.

So … now what you need to do is to temporarily load the tree, build and install the modules, and then remove the tree.

Sort of like cooking, where you need essence of kernel. But not the thing providing the essence as it would make the image too large.

This is the joy I am working through now. Works great BTW, in debian. Not so well in CentOS. Huge tree size.

The things we do to make end users lives easier and friction free …

Dealing with disappointment

In the last few years, I’ve had major disappointments professionally. The collapse of Scalable, some of the positively ridiculous things associated with the aftermath of that, none of which I’ve written about until they are over. Almost over, but not quite. Waiting for confirmation. My job search last year, and some of the disappointment associated with that.

Recently I’ve had different type of disappointments, without getting into details. The way I’ve dealt with these things in the past has been to try to understand if there was a conflict, what could I have done better. Regardless of whether I was in the right or the wrong, to try to look at things from a different viewpoint.

Dealing with the collapse of Scalable has been a challenge, personally and professionally. I can’t get into all the reasons why right now, as we are awaiting conclusions on a few things. Dealing with what came next, from the groups that thought they would cherry pick the remains, and tie me into their organizations on the cheap, was tremendously frustrating.

Value for me would have been help with relief of the personal monetary burden imposed. Not indentured servitude, which what was effectively proposed. Doing due diligence on some of the folks who reached out to me suggested that this would be their behavior. I naively hoped against hope that this time would be different, that their past behavior would not be reflected in dealings.

Yeah, I’ve got lots to write about there someday. This I am dealing with, and I’ll eventually write about it.

How to deal with disappointment … where there is conflict, I try to work with people to understand the nature of the conflict. I’ve found in the past that keyboards and asynchronous messaging systems tend to amplify contact, but personal contact, phone calls, enable you to more accurately express thoughts and listen.

As long as I am conversing with a person who wants to get to a point of resolution or even agree to disagree, I can manage. What I find disappointing is, when this is not the case. When the people involved seem more focused upon the argument, or a narrative that doesn’t mesh with reality. I take people to task for stuff like that. On my teams, on other teams.

As Mark Twain may have once written

A lie can travel halfway around the world before the truth can get its boots on

Information asymmetry is a tool of those who wish to manipulate. If you inject falsehoods into discussions, you can get your opponents to waste their energy countering a massive flow of accusations. This is basically how political agitprop channels work here in the US.

It is sometimes called “throw stuff at the wall and see what sticks” but in parallel, with many throwers. Usually done to protect ideologies. The instigators of such know they are on shaky ground. Many are not thinking big picture, and “be careful what you wish for, you just might get it.”

I try to turn conflict into a way to improve myself. This way, disappointment that a conflict exists become a mechanism for self reflection.

But when that conflict is being seeded with nefarious motives, and people I otherwise genuinely respect and like are actively engaged in it … yeah, not so much I can do there other than to disengage, let their fire cool.

Later on, when they are ready to re-engage, I can decide if I really should or not, as past unprofessional behavior may indicate significant issues with potential future unprofessional behavior. I am too old for drama, and intrigue. I don’t have time to waste on stupid arguments, and ridiculous narratives.

Attack the premise. Not the person. Reflect deeply before attacking the person. Accusations have a way of metastitising. Asynchronous message systems have a positive feedback loop on these, while phone calls have a negative feedback loop.

Someone offers to call you to try to understand your issues, take em up on it. Chances are they’ve dealt with bad stuff before and know that this is the better route to take.

Late Feb 2018 update

Again, many apologies over the low posting frequency. Several things that are nearing completion (hopefully soon) that I want finalized first.

That said, the major news is that this site is now on a much improved server and network. I’ve switched from Comcast Business to WOW business. So far, much better speed, more consistent performance, far lower cost per bandwidth.

I do have lots to write about, and have been saving things up until after this particular objective is met, so I can work/write distraction free.

More soon.

Apologies on the slow posting rate

Many things are going on simultaneously right now, and I have little time to compose thoughts for the blog. I anticipate a bit of a letup in the next week or two as the year comes to a close.

Cool bug on upgrade (not)

WordPress is an interesting beast. Spent hours working through issues that I shouldn’t have needed to on an upgrade, as some functions were deprecated.

In an interesting way. By removing them, and throwing an error. Which I found only through looking at a specific log.

So out goes that plugin. And the site is back.