You may notice they aren't called "Fun size sharing keys" or "Family pack keys" but instead "Private keys" because of that word "Private".
You don't need to wait for people to "exit the company". Sharing private keys was wrong, invalidate those keys. If somebody else knows your private key it isn't private any more. Get this stuff right and rotating keys is unecessary, get it wrong and rotating keys can't help you.