In the same way "colored people" can gain these connotations, just from other few people (falsely or not) inferring that it has those connotations. There need not be a history. I've seen too many blowups over the years about the word niggardly to think otherwise (more than one of these has made national news in the last few decades).
It's not that there is a history of discrimination, it's that we've all made a public sport out of demonstrating how not-racist we are, and people are constantly trying to invent new strategies to qualify for the world championships.
