This is only part of the picture. Part of the reason houses are so much larger today is cultural, but I would argue a larger part is that economies of scale scale make it more profitable to build larger houses, so that's what developers build.
Also, the average cost of raw, undeveloped land in 1970 was about $197/acre, or about $1690/acre in today's dollars. Today the average cost of raw, undeveloped land is about $4,350/acre.
All of the other costs associated with building a house are higher as well. There are more regulations, materials are more expensive, etc. Labor is probably the only part of the equation that isn't much different after adjusting for inflation.
So it's not just that we all decided to buy larger houses and that's why they are more expensive. There are a lot of structural issues at play.