Phoenix[0] has some good ideas about how X11 could be made more secure without breaking backwards compatibility. I don't understand what was so fundamentally broken about X11 as a protocol that it required a replacement protocol.
We can argue about limitations of X.org's implementation of the X server, but, as demonstrated by Phoenix, X.org doesn't have to be the only X server implementation.
[0]: https://git.dec05eba.com/phoenix