Perhaps I am being too literal here. But plenty of men are men, and yet are not taken seriously. So merely being a man does not mean you are taken seriously. In other words, no, men are not taken seriously "just because they are men".
Perhaps it would help to think of it this way: Most men are accorded a neutral rating from the get go and proving themselves puts them ahead of that. Women tend to start with a negative rating and have to work their way up to neutral. Yes, if a man screws up he can drop into the negatives. But he is much less likely to start there.
