I agree with your point.
However, the reason that I said "men" is that most of the time women aren't the problem. There are three main scenarios where condoms are helpful to society:
1) As a form of birth control. In this case the woman may not have access to other forms, and she has to rely on the man wearing a condom. If the man doesn't like it she is often left without options (particularly in jurisdictions where - unfortunately - consent laws are different for women who are married). Even if a women doesn't like sex with a condom she usually prefers it to yet-another-baby.
2) As a protection against sexually transmitted disease. A subset of that is sexually promiscuous women who dislike condoms (and in that case you are 100% correct - a condom which feels good to women will help). However, the other case - women who are paid to have sex - is unfortunately more common, and in that case the woman often has few options if the man doesn't want to wear a condom.
3) Gay Sex. Obviously this is really a subset of (2) above, but I'm putting it separately because it's quite a large vector for sexually transmitted diseases, and increased condom use would cut it dramatically.