Neither bad regulations nor the free market are going to solve a lack of foresight. Worse, those regulations are largely devised by politicians in power voted by a majority of homeowners and a few ultra rich lobbyists. You might as well make a "always has been" meme out of it.
Same goes for every other of your points. If free market capitalism would solve it, it already would have.
For example you decry the " inefficient space usage " of modern housing, and suburbs, however many people clearly prefer to have single family structures on plots of land over living in densely populated area's
I am one of those people, I would never choose to voluntary live in a dense city like NYC or even LA
I can think of nothing more nightmarish then living in a multistory apartment building. I lived in a 3 story building once, and got out of there as fast as I possibly could. Today I live in a typical suburban neighborhood, with homes on 1/2 to 3/4 acre lots and about 40-100 foot separation between home even that is far too tightly packed for me, and would prefer even greater separation.
Governments attempt to control sprawl and force more density is in part what is driving the massive increase in cost for Single Family homes.
So it is not a correct statement to proclaim the free market is not solving problems, it is that people that guide the market reject your proposed problems and the solutions you desire for the problems, in that rejection the only way to do it is to impose government regulations by force.
why is that? bad neighbors?
maybe the problem is our culture of selfishness and lack of tolerance and respect? (and by that i mean your neighbors that are causing you trouble, not asking you to tolerate that trouble)
i have lived in multi story buildings all my life in different countries. and never had issues with neighbors. i hardly ever see them. really, the only difference between an apartment and a single family house to me was access to a garden. and in a single family house i have to deal with neighbors too. so i really don't understand what your problem is there.
people have preferences. sure. my preference is to live close to shopping and culture, and my friends, and access to public transport without needing a car to go anywhere.
if i wanted to get away from neighbors i would move to the countryside where the next neighbor is a mile away. but that is no longer a city. suburban life to me is the worst of both worlds. (and i have lived that too, so i speak from experience on both)
the reason the free market does not solve problems is that the free market is not fair. in a free market the strong have an advantage at the expense of the weak.
slums are caused by a free market. to eliminate poverty, intervention is necessary. if an intervention fails then that's because maybe it is the wrong kind of intervention. and with that we are at the problem of figuring out how to make life better for everyone, which is a question of education, the will to make things better for others, and again respect and tolerance and giving everyone a voice, aka democracy.
The closest I can think of is Tokyo, where the zoning laws are basically set by the national government. In the USA, the local landed gentry can easily block any new development/increase in density, which is a big part of the reason we have so much sprawl*. In Japan, all you can do is shake your fist and write an angry letter to your congressman while developers throw up a 4 story apartment complex next door.
The result is that you can get a job as a janitor in the heart of downtown, and comfortably afford an apartment nearby. A tiny apartment within 10 minutes, or a larger and more spacious one 20-30 minutes away, or a house if you're willing to put up with a 90 minute commute.
* Yes, I know that a lot of the sprawl comes from people wanting houses, but a lot of people just want a place to live close to their job, where they can change out their sink or repaint their walls without asking some rich prick for permission
but look at any city with a growing population.
vienna for example is growing rapidly in the last two decades, after a slow decline of its population for almost a century, and it's interesting how the city addresses that growth with sufficiently dense housing. a whole new suburb is being built on a green field (literally) for 25,000 people and 20,000 jobs (not a bedroom community where everyone is commuting into the city), and that addresses only a fraction of the growth so elsewhere the city must be growing in density too.
Of course, the kinds of apartment buildings allowed in the US aren't that good either, since few cities allow single-stair buildings like Europe does.
Sprawl is a government invention, as is suburbia. Residential zoning isn't a natural concept and didn't exist before post-WW2.
