Do you know OP personally? Do you really think it's reasonable to assume that everyone in the universe (except for you, perhaps) is a hypocrite like this?
There's plenty of people that feel the administrative force of the university shouldn't be used to suppress either side. Let the gun club invite Luigi. Let the trans club invite the Stonewall rioters.
You're welcome to say you dislike the speaker. You don't have to attend. But you shouldn't have the authority to stop other people from inviting them to speak, or to stop other people from listening.