Speculation is also a problem when you have to pay to play.
Everyone needs housing, but when owning a house means it has a speculative resale value, you have to pay that premium in order to buy one.
This has a few consequences:
1. Downpayments and mortgages grow, because you aren't just buying shelter, you are buying a financial instrument with a significant resale value.
2. Your financial well-being is now deeply entwined in the fate of the housing and mortgage market. Higher interest rates become horrible for you, as they depress housing prices, and put your home underwater. New construction is horrible to you, as it reduces the value of the speculative portion of your purchase. Likewise for the existence of affordable forms of housing, and renter-friendly legislature.
Speculation from the perspective of a developer is part of running a business. They shoulder a fair amount of financial risk when they choose to develop, and obviously want to hedge it. Speculation as a 'every homeowner in the country is also a speculator, whether they want to be one or not' leads to some questionable outcomes.