It's complex but the reductionist explanation is that it is in part due to the euphemism treadmill, and in part etymological origin. "Colored" comes from the era of segregation, while "people of color" comes after.
But don't get too comfortable, as "people of color" will probably phase out soon as well. My money is first on "BIPOC" (as an initialism, usually sounding out the letters), giving way to "bipoc" (as an acronym, lower case, pronounced /baı-pok/). But I'm too white to make that call, just a guess based on linguistics patterns.
Even "BIPOC" is not without controversy. But afaik, no one has proffered an adequate word to describe "folks that have too much melanin to be considered white".
https://slate.com/human-interest/2016/08/colored-person-vers...
https://www.npr.org/2020/09/29/918418825/is-it-time-to-say-r...