https://www.health.harvard.edu/heart-health/sugar-substitute...
There's a good list here: https://www.efsa.europa.eu/sites/default/files/event/documen...
Given the glut of well-documented harmful effects of sugar, I'll take the "kinda sorta maybe" of aspartame any day, because I'm not about to give up every pleasure in life just because they all seem to kill me.
Thing is, if by 'they' you mean processed foods, you are basically right.
A little bit of sugar that you add to a coffee or whatever won't kill you. The problem is that just about all the processed food available has been spiked with sugar.