Only problem with that is that it's really not true.
ubuntu-10.10-desktop-amd64.iso: 695M
ubuntu-10.10-server-amd64.iso: 642M
natty-desktop-amd64.iso: 681M
natty-server-amd64.iso: 653M
Not sure I would call that "substantially different". Every Ubuntu ISO is a full CD's size, give or take.
Your reasoning for the difference isn't true either. X and X apps are most certainly included in the server ISO. Certainly, there's probably a few more packing the desktop ISO that aren't on the server one, but we're talking very little difference here.