Basically everything would stop working with X, because X simply relies on being able to listen to everything. Actually there are nested xservers that do something like this, but now global hotkeys don’t work, it doesn’t have an api for screenshots, so those won’t work either and the like.

And you can probably add some X extension which can’t be queried properly, but then you can just as well create a new display protocol that actually knows about GPUs