I saw a video where as a stunt a residential construction crew went from a vacant lot to a complete single-family house ready to occupy in less than two days. And that wasn't a prefab house, it was a regular wooden frame suburban house built using all the usual construction methods. They did it by staging all the materials right there and having all of the carpenters, roofers, painters, electricians, plumbers, etc standing around ready to jump in as soon as they were needed. Granted that was a small project, but the point is that with a sense of urgency construction can proceed quickly, and it doesn't require sacrificing worker safety.

I assume it was a limited number of people how knew how to make things and they kept roaming around setting new sites etc. Similar to bridge engineers etc. most of what they make just disappears in the background but they keep building things that makes our modern life possible.