UNIX sockets are perfectly fine for IPC with small amounts of data, and is how everything in UNIX has always done it, network transparency or not. They provide a simple, efficient and reliable communication channel between two processes.
1. You don't need to convert your discrete messages into a stream with size metadata, only for them to immediately be converted to a message on the other side.
2. You don't need to jump into the kernel to copy over 20 bytes, only for the other side to jump into the kernel to copy it back.
3. You don't need to deal with the "oh but what if my read returns half a message because this is a stream"
4. You don't need to pretend you're doing network programming.
It's pretty representative of the project though - "We're doing things the way we've always done them, but slightly different. Now rewrite all your software to work with our thing. No, you cannot do global keyboard shortcuts or set window position. You don't like it? We're doing this for free, you cannot critique it."
You need some kind of serialisation either way. It can be as simple as “this message has the shape of this C struct”, but that’s the case whether you’re talking shared memory command buffers or sending data over a socket (and there are good arguments for and against in both cases).
You’re right that you don’t need to deal with “oh I received half a message” when using shared memory command buffers, but that’s more a code complexity thing someone solves once in wayland-client and then nobody has to really think about it again. It’s not really a performance concern (because hopefully the rx buffer is large enough for it to happen rarely) or application code complexity concern.
If the compositor wants to notify the client as soon as possible, it has to send 1000 messages per second. If you buffer them, you're wasting the hardware's potential, if you don't buffer, them you're doing 1000 write()s per second, which is... ugh.
If you're literally going to design the protocol from scratch and require all existing software to deal with it, why not pick the IPC model that doesn't have this issue.
Sounds like a security nightmare.
A long time ago when I looked at designing a X11 replacement, that was my approach. AFAIK, only special X utilities used anything but Xlib anyway. And later I think this is what early revisions of Canonical's Mir did.