Lots of people want to design their own homes. For them, paying up for an empty lot is worth it. Put another way: those buying empty lots are developers or rich. With rates rising, the latter set marginal pricing in desirable places.

So no, there is no teardown required. But a teardown is desirable to some people. They bid up empty lots. Hence the deviance you notice, which is well documented in other places rich people like building designer homes. (Hamptons, Jackson Hole, et cetera.)