Like it or not, perception matters. Take accents as a similar and more obvious case: rural accents will often cause you to be looked down on, and so some find it extremely valuable to deliberately adjust their accent (in some lines of work, it’s pretty much essential to hone a certain accent). So it’s often worth at least
contemplating such feedback, if you care about being taken seriously. (But don’t get too hung up on it if you’re just a normal person, either.)
Just to emphasise this the more clearly: I’m talking about facts of life here, not ideals.