There _is_ harm. At one of the places where I worked previously, we used a few dedicated servers in different cities, and periodically synchronized data from the central one to all others using rsync-over-ssh.
Sometimes we got a warning from rsync that the connection was unexpectedly closed. We have traced this warning to SSH credential bruteforcers (yes, completely futile) that exhausted MaxStartups. So, we installed fail2ban.
I've ran into this myself. I couldn't ssh into my server until I disconnected the router (home server, and I was in the LAN). Turns out it was an extreme case of ssh bruteforce attempts that maxed out connection count or maxstartups or something like that. Don't remember exactly which resource was the bottleneck.
