Then if you need to get in, download the key, type in the password, login as that user, and the su up to root by entering the password only you know.
As an additional layer of security you could encrypt the key using AES-256 (e.g. 7Zip archive, Microsoft Office's Word .docx (NOT .doc) format (just drag drop the file into Word, "Encrypt with password") now AES-256 encrypted, others).
Now you have four layers of protection:
- su password.
- Key password.
- AES-256 encryption password.
- "Cloud" storage credentials.