Reality is not idealism. It would be ideal if people didn't have to care what they were wearing and I don't think people disagree with "that would be a better world".
The keywords are "that would be a better world" compared to "this is the world we live in".
You don't see people going into a ghetto at night wearing a Rolex watch, a 20kt gold chain, a money clip in their suit pocket, and another wad of cash sticking out of their back pocket.
Those sort of people, the kind ignorant of reality, end up being mugged or robbed. While nobody thinks "they deserve to be robbed" people will say "what the fuck was he doing walking into a ghetto wearing $25,000 in jewelry? Of course he was going to get robbed."
Recognizing reality for what it is instead of what people wish it to be is different than victim blaming. People will try to conflate the former for the latter though.