I do the same thing (plug both computers directly into different inputs on the monitor but use a USB switch to switch the mouse & keyboard). If you install this software on both computers, you can eliminate the "switching via OSD" step, because it will watch for the USB connect / disconnect events and initiate the input switch automatically via the DDC connection to the monitors, so it gives you true one-touch KVM switching at the cost of a USB switch.

This works especially well in cases where an actual KVM switch would be expensive or unavailable (in my case, I'm switching three 4K HDR DisplayPort monitors, which would require a very expensive KVM switch).

https://github.com/haimgel/display-switch