Is it really
much harder to leak a private key than a passphrase? (It's obviously harder, but not sure whenever a difference is significant.)
While one can't peek from behind your shoulder, if they got a keylogger on your machine, they could steal ~/.ssh/id_* files as well (and sniff their security passphrases too).