Reality says it depends. Various distros load various utilities. Kubuntu and Xubuntu for example use close to the same amount of memory from on button press to sitting idle. Thus it all depends on what you launch. It's quite common for a browser with lots of windows to use 8-10 times the memory that either xfce or kde uses at start so it really is diminishing returns if you have a modern system with 8+ G of ram.