Everyone is free to their own opinions. Actions can and should be banned depending on their motivations. We already apply this to other forms of restriction on speech like defamation. Often defamation will require negligence to harm or intent to harm. Someone who is factually incorrect through an honest mistake generally isn't considered to have committed defamation. I don't know why we are okay trying to ascertain the motive behind speech in one instance and not another.