Libwayland is a small library carrying the implementation of the wire protocol, and a few other bits like a simple event loop for servers and a library that can load X cursors. The point with that is that you're bringing your own compositing and multiplexing anyway. From there it's optional if implementations want to put in additional features for IPC, window management, hotkeys, screenshots, etc, and they can choose if they want to put that in the server or put it in a separate program. So you can still mix and match on some level anyway, it's not quite the same though.

I had exactly the same idea as you a few years ago to build something like that and I thought it would be useful too, and I thought about it for a while and realized that it doesn't really give you any of the benefits of Wayland or the benefits of X11. The point with wayland is already that it strips the unnecessary bits out and maintains a legacy code path with XWayland, and the point with X is that it's always going to keep the legacy code running anyway, so you don't gain much by combining them. If you have a minimalist window manager that's only a few thousand lines of code, and you want to get the benefits of Wayland, it's much easier to just port that using wlroots or something than it would be to rewrite the whole X server. That's just my experience.

>Sounds like a problem with the particular extension, not X11.

Since this is an issue with Xorg lacking the right extensions it's basically the same thing.