Oh, THAT'S what you mean by "crashing". I thought either the guest kernel (ie, vmlinuz or ntoskrnl) or the VM (qemu/KVM) was at fault, given that description and context. Thanks very much for the clarification.
Does sound like a graphics acceleration problem, maybe also a remoting protocol issue.
I only tend to launch VMs via qemu-system-x86_64 directly, so I'm not familiar with how virt-manager works - and crashes :). It sounds like virt-manager can crash without taking down associated VMs...? I'm also interpreting "start another instance" as referring to virt-viewer, or do you mean you're starting a new copy of virt-manager over the same/old VM?
It's really perplexing this also happens with VNC.
I wonder what would happen if you enabled the normal GTK qemu window (which for reference can of course be run in Xvnc in headless scenarios, my personal Xvnc preference being TigerVNC). I also wonder what would happen if you enabled SPICE and/or VNC with the GTK window enabled.
A better start point likely to produce more interesting/orientating info might be to run Firefox inside eg an openbox session, making absolutely sure no compositors are running (including xcompmgr etc). If you still get crashes in that type of scenario something's very broken.
I'm curious what error messages appear in the virt-manager, virt-viewer and qemu error log files when these disconnects occur.
It I were serious about fixing this my first step would be building everything from source, one component at a time, verifying the issue is still present at each step; then, if everything's still crashing, pulling out good ol' printf. Ideally something obvious emerges before you get to that point :)