The beauty of SSH-only is that you can assume that all of your traffic is being inspected all the time, but you have a protection against that: ssh-encryption and key fingerprints.

If you wanted to confirm ssh host-key validity, I'm sure rsync.net would perform an out-of-band verification. When they emailed me a request to do some server maintenance, I asked for a verification, and they placed a GPG-signed confirmation on their web-server for me to verify.