Know what else is artificial? Insulin and penicillin.
Even if they may have side effects/allergies, we tolerate them because they provide extremely large benefits to the population. You can't compare that to a chemical we use to make the colors of candies pop more.
No one is dying or getting seriously ill because their Fruit Loops had bland and unsaturated colors.
The answer to the above question is not a scientific one. It has to do with how we want to operate as a society, it’s a political or social issue.
When people say this they are obviously not referring the the definition of "chemical" that a chemist would use. Pretending otherwise is exactly the "nerd-snark" mentioned above which makes people distrust experts because they clearly aren't intending to use the term "chemical" in a sense that would include substances like water.