X.org is a display protocol, its own server implementation of that protocol, and many utilities to interface with that server. And even more stuff.
To illustrate this: if you run Gnome on X.org, you will see you have an 'xorg' binary running in addition to all the Gnome stuff. That's the X11 server from X.org. If you run Gnome on Wayland, there's no 'wayland' stuff running. There's just Gnome implementing the Wayland display protocol.
That's why this "switching to Wayland" talk is just missing the point. You're not really switching to Wayland, you're just switching to a "pure Gnome" stack, or a "pure KDE" stack, or a "pure Sway" stack. These stacks all just happen to implement the Wayland protocol in their own compositors (i.e. Mutter, KWin, Sway) because they're committed to standardization and all want to run GTK3+Qt5+Ozone+XWayland+etc apps built for Wayland.