Sort of in agreement with you, but on this, I think there's good reason to be concerned with something like this. If people with social influence disagree with your position and are public about it, they are in a position to influence more people than you can, and your position is less likely to be heard/adopted/enacted/etc.
I'm not sure that's actually why people get upset/involved when famous people hold opposing views - I think it's probably something more visceral - but I can say I don't want views in opposition to mine to get amplified favorable treatment in the media.
Perhaps secondarily, people get upset with famous folks holding opposing views because it ruins our perceptions of them. We often think we 'know' famous people on some level, and when they break out of the model we have of them, it's bothersome.