The owner might have a different opinion.
> Your concept of harm reminds me of how some people think they are "harmed" if someone doesn't want to date them
...and? That's a pretty commonly recognized harm. For example, it's the basis of the tort "alienation of affections".