You could use challenge response with the yubi (or just oath) and save the ssh key passphrase in keychain.. or you could get a yubikey neo and store the ssh key in the yubikey

neos can store the rsa key and use it to authenticate via ssh, as it has opengpg smartcard emulation support