There are consequences that can follow in response to protected actions that don't rise to the level of prohibited retribution.
For example I can give a speech in a public square where I advocate some completely stupid conspiracy theory and I do it in the most offensive language possible pissing off everyone who hears, and be protected by the First Amendment.
That doesn't stop you from inferring from that speech that (1) I'm an idiot and (2) I'm a very unpleasant person to be around and then based on those inferences declining to hire me if I apply to you for a job. Neither idiots nor assholes are protected classes so you are free to discriminate against me. That you learned that I'm an idiot and asshole through my First Amendment protected speech shouldn't be relevant.
If someone lets it be known who they voted for and their reasons something similar could happen.