There are many cases where an attacker can access a device for a short time and/or without the owner realizing that the phone was tampered with.
If you leave cryptographic keys lying around unprotected they should be assumed to be compromised.