How do you know it's actually worse for remote? I can render my entire UI on the GPU & route that straight into a video encoder incredibly quickly. Faster than XLib can draw in the first place.

Now there's the bandwidth question there yeah. The video stream takes X Mbps, and the Xlib stream takes Y Mbps. Unless you've measured it recently, you don't actually know if X or Y is smaller. You can make a guess that xlibs is smaller, sure. After all it's vector, right? Except it isn't, not really. Text is all pixmaps. Images are all pixmaps. Path clipping is all pixmaps. Are you compressing all those pixmaps before sending? If so, to what? And is that faster than your GPUs h264 encoder? Almost certainly not. Is it smaller? Also probably not really.

And I don't know about you, but I got way more bandwidth than I have latency for my Internet connection. So whether or not X or Y is smaller doesn't actually matter to me. What matters is which gives me less end to end latency, which all but certainly is the video encoder & decoder path, which are incredibly well optimized and hardware accelerated.