Those are my chief complaints as well, actually. I never quite got to the point where I grasped how all the bits fit together. I understand the DSL (though the errors are cryptic as you said) and the flakes seemed recommended by everyone yet felt like an addon that was forgotten about (you needed to turn them on through some experimental flag IIRC?).

I'll give it another shot some day, maybe it'll finally make sense.

>Oh man, having a 25U rack sounds really fun. I have a moderate size cabinet I keep my server, desktop, a UPS, 10Gig switch, and my little fanless Home Assistant box. What's yours look like?

* 2 UPSes (one for networking one for compute + storage)

* a JBOD with about 400TB raw in ZFS RAID10

* a little intertech case with a supermicro board running TrueNAS (that connects to the JBOD)

* 3 to 6 NUCs depending on the usage, all running Talos, rook-ceph cluster on the NVMEs, all NUCs have a Sonnet Solo 10G Thunderbolt NIC

* 10 Gig unifi networking and a UDM Pro

* misc other stuff like a zima blade, a pikvm, shelves, fans, ISP modem, etc

I'm not necessarily thinking about downsizing but the NUCs have been acting up and I've gotten tired of replacing them or their drives so I thought I'd maybe build a new machine to rule them all in terms of compute and if I only want one host then k8s starts making less sense. Mini PCs are fine if you don't push them to the brim like I do.

I'm a professional k8s engineer I guess, so on the software side most of this comes naturally at this point.

For certain definitions of the word “fun,” yes. I have a 35U (I don’t need that many slots, but at the time I did need it tall enough that my kids couldn’t reach the top, where I put the keys), with:

* 3x Dell R620

* 2x Supermicro (one X9, one X11)

* 1x APC UPS w/ external battery

* Unifi UDM Pro

* Unifi Enterprise 24-port switch

The Dells have Samsung PM863 NVMe drives which are used by Ceph (managed by Proxmox), with traffic sent over an Infiniband mesh network via Mellanox ConnectX3-Pro.

The Dells run K3OS in a VM, which is a dead project. Big mistake there.

The Supermicros have various spinners, and are in a ZFS pool. One of them is technically a backup that should power up daily to ingest snapshots, then power off, but there’s been some issue preventing that, so…

It was all very fun to set up, and has been eminently reliable, but it’s a bit much. While you can in fact make R620s relatively quiet, they’re still 1U, and those little 40mm fans are gonna whine. It’s background noise to me, but guests definitely mention it if we’re close to the rack.

Also, I’m now in the uncomfortable position of being stuck on Proxmox 7, because v8 (or more accurately, the underlying Debian release) dropped support for my HBAs, so the NAS would be dead in the water. I mean, I could compile my own kernel, or perhaps leverage DKMS, but that somewhat defeats the purpose of having a nice AIO like Proxmox. Similarly, my choose of K3OS means at some point I need to spend the time to rip everything out and start over with Talos.

Or - just maybe - I’ve done enough playing, and I should simply buy a JBOD chassis and a relatively new and quiet server (4U under light load means you can get away with much quieter fans), and just run stuff in Docker or gasp systemd. Or, hell, single-node K8s. My point is that it is fun, but eventually your day job becomes exhausting and you tire of troubleshooting all the various jank accumulating at home, and you stop caring about most of it.