There are more
components to housing prices than just not building enough, but in ordinary markets that aren't constrained by pathological factors (e.g. China), supply and demand operates and increasing supply lowers prices.
For example, suppose you don't have artificial supply constraints and then you lower interest rates. No problem, more people can get a loan to buy a house which causes more houses to be built and them more people get to be homeowners. But long-term prices stay the same, because it's easy to add supply, so it keeps getting added until prices fall to the construction costs, which acts as a long-term ceiling on prices.
Now suppose you do have artificial supply constraints. Prices are at $200,000 but you've made it cost $1,000,000 to create a new housing unit. Now if you lower interest rates, prices are going to go up, because you can't increase supply at the current price so people just end up having to outbid each other on the existing supply. And the same thing if local demand increases or anything else like that, because you've prevented supply from responding to demand until prices hit a million dollars.
But the problem isn't low interest rates or growing cities or anything like that, because the only reason those things are a problem is that supply can't increase until prices are unaffordable.