I'm not saying anything should be taken away from anyone. Just taxed. Taxed enough to encourage you to do something with the property you sit on. Like ensuring you rent it at full capacity or sell it if it's not worth renting (because of tax).
Tax is so low right now that you can sit on whole lot of properties for years and not even bother to rent them and still not loose much money.
That seriously harms others and should be at least effectively discouraged.