Presumably you sign the emails rather than encrypt them?
Otherwise anyone who knew the public key of the server (which shouldn't be presumed secret) could send an encrypted instruction, and it would be acted upon, and past encrypted instructions could be replayed.
> Presumably you sign the emails rather than encrypt them?
That's correct, encrypted and signed. Replaying wouldn't be easy because the payload contains a timestamp. The main purpose was to limit the networks which can attempt to connect to ssh and still allow me to have a fallback if I'd happen to be outside of the "usual" network ranges.