I have used strictly Nvidia gpus, on debian Linux, for 20 years, using the binary drivers, and I have none of the issues you describe. It just works, very well, stable and fast. Xorg is an amazing piece of software and combined with openbox it does exactly what I need.