After doing a bunch of research I eventually bought a monitor with a KVM built in and I’ve had almost no issues with it. It does imply if you go this route that you are stuck with a single monitor setup, but if that is how you work anyway it’s totally fine and great.

Otherwise some KVM that is decent is as you imply not going to be cheap. The one that I have seen recommended on here before is https://www.store.level1techs.com/products/p/14-kvm-switch-d...

The manufacturer themselves even outright says it’s not perfect, but from what I can tell it’s probably the best one on the market right now