You can put a passphrase on the key and not worry about someone else getting it by stealing your phone, just remove it from authorized keys when you're back at a computer. Unless you're concerned about needing emergency server access while your phone got stolen and hasn't been replaced yet.
But at that point, we're probably also worried about the odds of getting run over by a bus and having amnesia that prevents you from using a password protected key, chance of getting struck by lightning, etc. ;)
