Vacant housing is only a problem in constrained areas. The vast majority of the U.S. is not constrained. Your summer cabin in Montana isn't depriving anyone of a home, because it isn't driving up prices. Your unoccupied condo in Manhattan is depriving someone of a home, but I suspect that there are not so many of these.