At least in the US there never was widespread slavery of white people because they are white, or widespread ideas of white skin color being associated with lesser intelligence, culture or whatever.
It is, however, very important not to get hung up in racist terminology. For example it only serves the racists' cause to argue who is actually black or who is actually white, or how much of each.
It's not like the average heckler on the street asks somebody where he is from, his religion, what he is doing for a living or what his ancestral gene mixture is!
