If you need to access a server you'll always need a computer. authorized_keys are public keys only. it does not matter if other computers have your public key. all it does is give you access.

the private key of your ssh key(s) give access to several servers, thats why its the part that you want to protect. if one server has a rootkit, well, that sucks. but if that rootkited server can access all the servers YOU can access, you're screwed.