The air above your garden is not exactly 'public space'.
People will put all kinds of visible barriers around their house to get some privacy, hovering your drone over those barriers is clearly a breach of sorts.
Depending on where this is happening the laws will vary but here is one example from the UK:
Public airspace starts rather high (500 feet?), a random drone flying over your garden at 300 ft height is technically located in your private property.
