It depends a bit on intent though - if the purpose of building the house is just to extort the other landowner of money, and it would make a meaningful difference to their house value, it starts to get a lot more grey to me.

There are a whole string of scenarios between “not blackmail”, to “probably not blackmail” to “definitely blackmail” depending on the scenario. A bit like how with assault if you poked someone that is probably fine, but that doesn’t make punching someone in the face ok (but what is a punch if not a bigger poke?).