Because land has fixed supply, as you've said, the tax is perfectly efficient: it causes zero deadweight loss. But LVT actually has positive externalities as well- namely, it makes it nearly impossible to speculate, which with land consists of buying land and holding it off the market unused or underused in order to sell it later. Speculation is very inefficient and also quite common, so removing it actually makes the economy better than it was before an LVT.
As for the tax being too high, if people started to vacate land that would mean the tax rate is over 100% of the land value (if it's at 100% people won't vacate, they won't be able to extract rents from it but it won't be an economic loss). Governments have every incentive not to tax land over 100% value because they actually lose money from doing so, so that's a pretty good security that it won't happen.
