Why shouldn't our neighbors use their powers as voters to keep their peaceful suburban street from being used as housing for migrants, minorities and other undesirables? I'd bet that'll affect housing desirability far more than short term rentals.
Why shouldn't they use their power as voters to ban the construction of affordable homes in their vicinity?
Why shouldn't we use our power as voters to proscribe alcohol in their town? What could go wrong?
Your power as a voter isn't supposed to be the power to arbitrary meddle in other people's business.