Regarding security, I'm honestly surprised no one has just tried to make it so you can "firewall" X11 programs from one another. Like, aren't keystrokes propagated as packets sent through an X11-owned UNIX domain socket in /tmp? Can't we just attach a policy to that socket to decide which PIDs (or process groups, session groups, containers, etc.) get to see which messages?
Most of the heavy-lifting is done in wlroots anyway. wlroots based compositors really do implement just their own flavour of compositing what you see on the screen on top.
That said, you still can use IPC, if you really want to; I have an external window manager that augments Sway's tiling system via its i3-compatible IPC mechanism to arrange my windows in a way that Sway doesn't do natively. If you really wanted to, there's nothing stopping you from writing a wayland compositor that uses an external window manager.
At any rate, I don't buy the reliability argument at all. I've used sway since 0.10 or something, and I only ever remember crashing it once, and I fixed that bug myself. :P
This can be done via firejail[1] + xpra/xephyr but is a rather cumbersome endeavor. The X11 standard also contains access control hooks that allow you to "firewall" any aspect of your application. However it is used by no application I personally know of and is rendered useless by how the xinput mechanism is implemented at this point.
The reason nobody bothered to deal with this so far is that people almost never run untrusted software on FOSS systems which is what X11 primarily targets. There was no demand.
1.: https://firejail.wordpress.com/documentation-2/x11-guide/
The response I've heard to this question is entirely nonsensical: it could be done with an X extension, but getting adoption from various parties to make this work would be difficult. As if building an entirely new display system doesn't require orders of magnitude more work and buy-in.
