Having a public key doesn't teach an observer your private key, and so they can't impersonate you, but it does allow the observer to distinguish you from others. If you would like to prevent observers from correlating identity this way (most famously if you use a public key for GitHub and also other things) you will want to explicitly forbid your SSH client from offering to prove identities other than the one you know will be used.
The setting in OpenSSH (which you can enable for individual Hosts) is IdentitiesOnly yes
Your proposed configuration will choose to try the file named, but it will not tell the remote server that you don't have any other identities, for that you need IdentitiesOnly. The default is no, although I guess it's possible you have overridden that to "Yes" previously and then forgotten.
