EDIT: Apparently, based on HN search, many HN users have realized the same within the past few years.
i'd go more with multi device keyboards and mice. like the logitech mx keys for example. have mine paired to 3 devices currently - easily toggle between them with a switch on the keyboard. Granted its not as seamless but sometimes i just want to quickly respond to an email on my personal rig while still watching work queues.
they also have mice/trackballs with the same tech - although the mxmaster mouse has the switch on the bottom so not as convienent.. and the mx ergo trackball only toggles between 2 devices.
other options if screen realestate isn't as important are things like the genki shadow play. While targeted at gamers - it works great as an HDMI input device. I use mine to work on raspberry pi projects on my main system. Its view only - won't pass control through it, but sometimes thats all you really want.
and on the other end of the spectrum, full display port KVM's like these from Level 1 techs:
https://store.level1techs.com/products/14-kvm-switch-quad-mo...
display port 1.4: 2 computers, 4 screen switching
Of course, you might run up into a problem that you actually don't want to plug some devices into the switch when switching to another PC (for example some audio equipment)... So in a dream device, there would be switches on-off switches for ports (or groups of ports) and target PC enum switch. Probably some modular approach would work for making groups of ports switchable.
EDIT: https://www.avaccess.com/eu/product-category/kvm-usb-extende... (link to the manufacturer's e-commerce site) seems to have a good repertoire of KVM extenders.
EDIT1: Bluetooth support is still the missing piece in these...
