That's not quite true. Every architecture iteration brought an increase in IPC, while every process improvement allowed for higher clocks and lower power dissipation. It's true that GPUs evolved a lot more - but that's mostly by getting bigger, wider, and having wider, faster pipes to its own RAM and the shared PCIe bus, as well as bigger heatsinks and fans.