Right now I am sitting at "my computer". On "my computer" I have a ton of running state: I have a bunch of open web browser windows with a ton of tabs in each, I have open terminal sessions that are currently displaying the output of histories I am interested in or are actively running programs that I'm working with... hell: I have background computations being managed by various programs, such as video editing and encoding software, that I am interacting with. You might choose to call this state my "desktop", and I want to have "remote desktop" access.
Now, I go somewhere else. While I'm out, I realize "oh, I need to change the queue on my encoding software" or I want to talk about some web page I have open. This is what Remote Desktop on Windows is doing for me: I don't connect to my computer remotely and then have some abstract notion of "I guess I can run software at home but see it here"... I actually am able to make that computer I'm at--wherever it is, and whatever it is (often it is my phone! the RDP clients for iOS and Android are excellent)--equivalent to the computer I have at home.
As far as I have ever understood--and I'm NOT an expert at X, though I've been using it for 25 years in various capacities ;P--X fundamentally doesn't do that: X clients connect to X servers, and if the X server dies then the X client also dies. The Window Manager--the behavior and interaction of which in this system I, in fact, care about deeply--also connects to the server. I have never come across a use case where I'd want to run a graphical app on one computer and merely have it display on a local computer: in that case, I can simply run that app locally.
However, what has given me superpowers is the ability to run the entire stack remotely and then "connect to it" from a local computer. I don't want to, while I'm out, run a new web browser process at home and have it appear on whatever device I'm at... I want to have the web browser I've already been running--along with all of its windows, all of their tabs, and their running state that have been actively continuing to execute and do stuff while I'm out--appear on the computer I'm at, making it feel everyone's computer becomes "my computer".