It's of course rude to insert yourself into many conversations. That's just a social convention.
> Avoiding women in workplace is essentially sexist. And for non-sexists there is no reason to do so.
So you're damned if you do and damned if you don't. Given how easy it is for public opinion to shift, I certainly wouldn't want to take that chance. Who would want to interact with people like that in a professional setting? I wouldn't, and I try not to.