If that makes the office more diverse, than the real issue is in the hiring bias, not in how you dress. For sure if you only hire people like you, which can happen both in a business environment or a social one, you will have an office less diverse, that's by definition.
I disagree entirely that we can't be social with people different as us. I also disagree that business shouldn't be social, but that's mostly an opinion.