Most of the heavy-lifting is done in wlroots anyway. wlroots based compositors really do implement just their own flavour of compositing what you see on the screen on top.
That said, you still can use IPC, if you really want to; I have an external window manager that augments Sway's tiling system via its i3-compatible IPC mechanism to arrange my windows in a way that Sway doesn't do natively. If you really wanted to, there's nothing stopping you from writing a wayland compositor that uses an external window manager.
At any rate, I don't buy the reliability argument at all. I've used sway since 0.10 or something, and I only ever remember crashing it once, and I fixed that bug myself. :P