I had a 4K Dell monitor as my main, a lower res monitor on one side, and a 16:10 in portrait mode on the other side, all serving two computers. The computer that used DisplayPort for the 4K monitor had the issue you mention; when I swapped source from one computer to the other my open windows all jumped around to the side monitors. Apparently it's the DisplayPort driver that's the problem; HDMI, DVI, and VGA don't behave this way, but DisplayPort does.
I've replaced the 4K Dell and side monitor with a big curved 8K Dell, which has a built-in KVM. My keyboard, mouse, headset, external speaker bar, and webcam are all plugged into the monitor, and they all swap from one PC to the other with a keyboard shortcut that also switches the monitor source. It's a lot handier. However, if I manually switch my portrait monitor to the PC that uses DisplayPort, my windows will still get moved to it when I switch everything else to the other PC. So I just don't switch that portrait monitor very often anymore. Using the keyboard shortcut has made my lazy and I don't bother with the last monitor unless I really need it.