I don't get this reference. Can you explain it or show an example?
There was a time where a popped collar was a sign that you were a problematic person. Probably still is in a certain context. I think people are going to do some pattern matching with your clothing and people they've met/seen in the past. It's just a bit of human nature.
Maybe everyone should dress like Mr. Rogers or something.