The first time you log into an SSH server, your connection is actually insecure (this is also true of every subsequent time you log in and see that yes/no dialog).
It's insecure in that you don't know the identify of the endpoint without verifying the fingerprint out of band. That problem is already solved in browsers thanks to SSL certs and certificate authorities.
Is there any way to be sure of an server's ssh fingerprint before logging in? You need to do that offline right? Maybe a PGP signed e-mail from the sysadmin?