Energy consumption is also about using that stuff. And in rich countries like the USA, a lot of energy goes to using things or just moving people. So it's possible to build more and use less energy, even if you don't reduce the energy cost of building things. They can simply become more efficient to use and move.
In USA, 30% is just homes and commercial sector. 40% is transportation. 30% is "industry". By a sort-of inverse-amdahl's law, it's possible to get lower energy use with more throughput even if we don't make "industry" more efficient.