Updating infrastructure

Decades ago, I had relied upon firewall devices which offered multi-wan capabilities, reasonable configuration complexity, and reasonable performance for the internet of the day. The day job had, at that time, a 100Mb download connection, and a 10Mb upload connection. Blazing fast by the standards of the preceding years of 16Mb down and 1Mb up. So my router needs were modest, and appliances based upon relatively low power chips/NICs would function well. More on this later.

Having worked at my home office/lab for the past 8 years, I needed (and still need to an extent) faster networking, for multiple/many clients. I've got a 5 node linux cluster (soon to be 6 node, and I need more power from my 200A feed, though that is a different story), many little edge wifi attached units (Roku, various phones, appliances, etc.). I've got 3 levels in my house, the basement where my office/lab (aka the "salt mines") and a downstairs living room are, the main level, and the upstairs where the bedrooms are.

We aren't using cable TV for our TVs, just wifi connected Roku. For the past 2 decades, I've used a 16 port 1GbE switch for all of the wired units. This changed about 2 years ago, where I started to use a 64 port Mellanox 10GbE switch purchased on ebay for about $200. This unit has 6x 40/56 GbE uplinks, and 2 are tied into my Mellanox 6036G IB switch also from ebay for about $300, where I'd bought the Ethernet licence, so I broke it out into 18x IB and 18x 40/56 GbE ports. This is mostly for my cluster and storage (28TB of ZFS on Linux based server).

My wifi has been various implementations of an AP mesh, or an AP backed by ethernet backhaul. More recently, over the last year, I started using 2.5GbE backhaul MOCA adapters connected with the APs. The APs have either 10GbE or 2.5GbE ports for backhaul, so I am leveraging those. I've finally settled on Asus routers for now for Wifi 6(+?). I don't have much Wifi 7 kit available right now, and likely won't for a while.

My router has been some variant of a 1 socket server for a while after using the appliances. I had deployed pfSense at first, as I was familiar with it from customer deployment in the early 2010s. However, after working with some people who eventually went to work at the parent company, and knowing their ... er ... engineering acumen, I decided to investigate and eventually transitioned to OpnSense.

I will admit I am disheartened by the dearth of equivalent linux based router OSes though the GL.iNet folks have been doing an awesome job with travel routers. Their system is based upon OpenWRT, though I've not (yet) figured out how to deploy this to include multi-wan, 10GbE NICs, etc.

So OpnSense it is. It had been deployed on an old 1U Supermicro, small server unit. 2x SSDs, 1x 2 port 1GbE NICs. That is, until that unit shat the bed. Again this was from eBay, E3-1230v1 processor and 16 GB ram. Maybe $200 when I bought it.

I replaced that with a proper 1U server. Another Supermicro, 4x disk slots with 2x SSDs. 1x Mellanox Connect-X3 card with 2 ports. Again that worked great, but it is bulky, noisy, and a bit of a heat generator. This was 2 years ago.

I did want to drop my power usage. I get regular notes from DTE, our power provider, over our utilization. I don't run the cluster 24x7, but it does seem we use a bit more power than our neighbors (if I am to believe DTE, which is an entirely different question).

So I noticed a deal on a Miniforum barebones PC that looked perfect for a smaller OpnSense machine. 2x 10GbE SFP+, 2x 2.5GbE RJ45, M.2 NVMe support, etc. I had extra ram from a laptop I upgraded (my Lenovo P16v) and a 512GB NVMe from another unit. I popped them in, loaded OpnSense and played with it. This worked quite well.

So yesterday, I backed up the other unit to an xml file, uploaded the xml file to the Minisforum unit, fixed the WAN and LAN port names, unplugged the old unit from the network ... and it was passing traffic!

I did some basic benchmark testing, and it seems the newer unit is faster at packet processing/handling. It is a 12 core 16 thread unit with an integrated intel Xe graphics unit. I don't need that part, but I did attach another JetKVM unit to that so I have console access. I just need a network addressable/controllable PDU for it. Soon.

Simultaneous to this, I am building some workstations for the day job. My daily driver machine is currently a Lenovo laptop, almost 2 years old. I'd rather use it as a laptop, as the USB4 docking station it is attached to occasionally hiccups, and then I'd lose display/network connectivity. This is annoying, and just goes to the point that docking stations aren't really meant for 24x7 connectivity, and laptops, while great tools, aren't really replacements for workstations.

One of the two workstations is a Ryzen 9 9950 X3D 16 core 32 thread for my main daily driver. It has 256 GB ram, a 2.5 GbE port, 1x PCIe gen 5 16 lane slot, 1x PCIe gen 4 16 lane slot, 1x M.2 Gen5 4x4, and 2x M.2 Gen4 4x4. I have a "spare" AMD Radeon RX7700 I plugged in for now, hopefully I can get work to supply a 9700 at some point. I put in a spare Mellanox Connect-X3 card with 2 ports.

I switched my office network to be mostly 10GbE RJ45 (10Gbase-T) apart from the few stubbornly 1GbE and 2.5 GbE NICs on their own switch. Machine to machine can be pretty fast, about 700 MB/s, so I'm happy with that. I'd love to get a quiet, 16-ish port RoCEv2 capable switch that didn't break the bank in the office, but then again, I am moving most of the compute gear to the "server" room portion of my basement.

The second workstation is waiting on some additional approvals. More about that later, but that is meant to be a local virtual cluster in a box, a build machine, a GPU monster, etc.

All of these updates have helped. None more so than disentangling 4 years of incremental networking connectivity. I had to revisit how my wifi connected to the 10GbE backbone/spine. I removed extra unneeded loops (yes, spanning tree wil help mitigate these), simplified conections between switches. Fixing all these things has made my wifi performance jump considerably. I now get close to GbE performance on my laptop's wifi. Which is what I wanted. Though I have a 2.5 and 10GbE adapter for it, I'm fine with 1GbE over wifi for the moment. I get that out to the internet as well.

Next near term objective will be to take my aging HP Procurve 16 port switch, and replace it with something with 16+ 1-2.5GbE ports and multiple 10+ GbE uplinks.

Longer term, I keep hoping against hope that someone will wire up fibre around me. AT&T claims to have 5Gb fibre close, and there is a POP about 1.5 km north. Metronet runs a fibre 100m behind my house. Ribblefiber made some noise about building up in this area.

The reason is that I want symmetric performance, and would prefer 2Gb or greater. I can use it (think large models/data sets being moved all the time). My upload speed is 5MB/s.

Putting that in perspective. That is 200s or so for 1GB of data. For 1TB of data (e.g. backup sizes over the interwebs), we are talking 200,000 seconds, or a bit more than 2.5 days).

More competition is better for customers. My ISP dropped my monthy cost by $100 due to competition. I am hopeful that someone will realize that for professionals working from home, higher speed higher reliability service (mine is "business class", though I am not entirely sure what this means beyond supposedly getting a better QoS) is a market worth pursuing.

More later.