Storage and GPU poor

I have a number of machines here in my home lab. A mix of GPU types I thought would be reasonable for low level AI work, CPUs with 16-32 cores, and 128-256GB memory.

GPUs are mostly AMD with 12-24 GB VRAM, apart from the strix halo which has up to 120GB unified memory.

Having built most of these nodes over the last few years, I gave them a generous 1-2 TB of NVMe storage.

I use lemonade and ollama, with lemonade calling out to llama.cpp. I can't seeGbm to get vllm working on my kit. I might need to pull a container for that.

Now the models and design choices of Ollama and Lemonade server. Models that are usable on a 16GB unit like my personal R9070X are under 12 GB in size. This is quite restrictive. I recently bought a past EOL MI60 32GB ram, put on an air cooler, and am running that. I can run models up to 28GB on that reasonably well. Not as fast as the later model GPUs at the computation, but inference is dominated by memory bandwidth, and that MI60 has 1TB/s from HBM2 to the GPU.

The fastest inference I get is out of that machine and the strix halo. I named them Marvin (as in the robot from The Hitchhikers Guide to the Galaxy) for the MI60, and hal9k for the strix halo. They are connected with 10GbE, though I m looking how I can drop in a 40GbE/56Gb IB card and a second GPU into Marvin. This will require some case mods, and a new power supply. Both systems are running Ubuntu 26.04 with a 7.0.0-29 kernel.

But the NVMe. Thanks to the reallocation of RAM and NVMe production lines into HBM production lines to satisfy demand for AI systems, both RAM and NVMe are now ridiculously expensive. Marvin was bult on an older spare motherboard, and I bought 256GB DDR4 2133MHz memory for it. So I have enough RAM in that box, and I had some spare 4TB and 2TB NVMe, so I put those in.

Both Ollama and Lemonade store models effectively only on the / file system. Lemonade gives you an option to add paths, but it is brittle in a way that is disconcerting. The first time I added an additional storage path, I completely broke lemonade. I couldn't unbreak lemonade by unsetting that setting. I appealed to claude to help me out with this and wound up simply uninstalling it, and then re-installing it.

Ok, the strix halo has a 1TB / directory, and a 3TB /home. I can't store models in /home. This makes no sense. Once I have some time I'll go looking at the lemonade source, see if I can find the bug, and fix it. This one is serious, in that it makes it much harder to use the product with multiple models, as they eat up lots of space very quickly.

I set Marvin (the M60 machine) to have a 4TB / and 2TB /home based upon my experience with the other config. This was a wise decision.

I also use my personal deskside system hermes, with the R9070x, as an AI engine. As it turns out, I do not recommend using your display adapter (the one driving your GPU) as a computing engine, with the exception of the strix halo. This has resulted in crashed X servers, and inability to interact visually with the machine for intensive workloads.

I have a small cluster in the other part of my basement with ~3 high performance nodes, scruffy, fnog, and leela. All characters from Futurama. Nibbler got turned into Marvin when the motherboard/CPU died. All running on a Mellanox switch running both IB and ethernet.

But the GPUs in those systems are 8GB 12GB and 16GB. Mixture of consumer cards, and 1 MI50 with 16GB HBM. I can't as far as I know, run larger models across multiple machines with llama.cpp (please let me know in the comments if this is not true). I've heard about using the IPC mechanism, but haven't looked into it yet.

I really would like an MI350p or 2 (600W each ... eek!), or a set of MI210s, etc. I see NVidia V100s on ebay (where I bought most everything over the past few years), and they are comparible in price to the 32GB MI60.

Of course if I were to update the NVMe storage in these nodes to something more reasonable, it would cost more than the GPUs.

So I remain GPU and storage poor. Kimi-K3 is a 2T parameter model that clocks in around 1/2 TB. Never mind the lack of GPU with enough VRAM for that, I couldn't afford such a machine, and while many claim to enable hourly rental, demand is so high you are hard pressed to find any.

I didn't have being storage and GPU poor on my 2026 bingo card. But there it is. My personal work must run only on my personal systems, not on my employers systems. So I can't borrow them. I use those models enough during my normal work day, but it is quite frustrating having to step so far down in capability, and use diminished models.

I should also note that after all this experience, I think my next desktop/side system will be another unit like the strix halo, but I don't think I will buy that for a while.

For those suggesting simply running in the cloud, I've been doing that as well, and it is expensive. Even with the "pro" plan, I can blast through session, weekly, monthly, and quarterly allocations quite quickly. I can't really justify the $100/month and up plans.

Ah well. Maybe Santa will come by with a 350p or similar