I'm totally with you here. One should not try to get rid of these quick form of prejudices, because they exist anyways and they might be useful.
To make them less "dangerous" it's good to be aware of them and stay doubtful about them to give anyone a chance to proof you better.
Reminds me of how I handed my phone to a couple once and wasn't sure if they'd scam me, but I willingly took the risk of loosing my phone to be able to help people when they ask.
They where honestly quite sketchy... I still wonder if they maybe just spontaneously decided not to scam me because they felt I was prepared for it.