If you're not racist, being labeled racist can cause real, concrete damage. Economic damage, social damage, professional damage. In the current climate, it is to at least some degree rational to want to not be labeled racist.
No one would want to be labeled a racist. I think the difference is how constantly worried you are you about it — do you change your behavior in order to steer clear of anything that might remotely look questionable. My friends who are also coworkers never behaved differently around me when in public or in private.
