You can use ssh_config to limit which agent keys are offered to which remotes. https://superuser.com/questions/272465/using-multiple-ssh-pu...