Perhaps you mean to say that speakers are unable to name the difference between the colours?
I can easily see differences between (for example) different shades of red. But I can't name them other than "shade of red".
I do happen to subscribe to the Sapir-Whorf hypothesis, in the sense that I think the language you think in constrains your thoughts - but I don't think it is strong enough to prevent you from being able to see different colours.