This can be done via firejail[1] + xpra/xephyr but is a rather cumbersome endeavor. The X11 standard also contains access control hooks that allow you to "firewall" any aspect of your application. However it is used by no application I personally know of and is rendered useless by how the xinput mechanism is implemented at this point.
The reason nobody bothered to deal with this so far is that people almost never run untrusted software on FOSS systems which is what X11 primarily targets. There was no demand.
1.: https://firejail.wordpress.com/documentation-2/x11-guide/