This is the key point of this argument and it boils down to the idea that freedom of association is somehow less meaningful or more able to be limited than freedom of speech. It's not. Even in a business context.
> If you don't want to work with him, that does not merit him losing his job.
If I'm his employer, then yes, this merits losing his job because it's literally the definition of why people lose their jobs. Because people don't want to work with them. Whether that's due to things they say or things they've done is irrelevant.
The thing is, speech matters. You can't arbitrarily separate the world into "speech" and "actions". The nursery rhyme about sticks and stones is incredibly untrue. Speech is the predecessor of actions and it tells you both what someone intends to do and what they want you to do.
If every single company in america signed some kind of agreement to never hire James Damore, or the federal government passed some kind of law forbidding his employment, then yes, that would be extreme and unjust. Instead he just got fired and had to interview at a new company. Hardly an existential crisis.
This again goes to my original point, Damore is in a fairly privileged position and didn't actually suffer that much, and yet we're supposed to use this as an example to justify silencing people.
The thing that gets frequently glossed over is that all of these situations where people are "cancelled" are merely reversions to a neutral position. Hiring someone, and by extension keeping them employed, is an action you take. Firing them is merely stopping that action. Same thing with inviting someone to come give a speech at your college or anything else. Cancelling the invitation is merely reverting to the original, neutral position where no action had been taken. It's not some kind of massive injustice if rescinds an invitation, no matter if that's to a party or to give a speech.
Gamergate is especially ironic since it was essentially an attempt to cancel someone that started all of it, it just turned out to be based on a ton of false accusations and then escalated into frankly criminal behaviour.
The whole purpose behind the "cancel culture" meme is an attempt to prevent people from reacting to speech. I think that's wrong and damn near evil. Speech can be incredibly impactful and being able to speak and act in opposition to it is sometimes the most important thing anyone can actually do.
Like most things in life, it turns out that why you're doing something actually matters quite a bit. There are people in this world who absolutely deserve to be "cancelled".