Why? Do you think a poor person's vote is worth more to the government than their wallet would be to a house builder?
Is it impossible to make an acceptable house at 30% the cost of a median earner's house?
Your reasoning is valid for people whose productivity approaches zero, in which case welfare can indeed be needed. But the current housing problem is systematic and touches a far bigger percent of the population, and thus shouldn't be solved with charity.