Sugar is not the enemy, sugars are just a highly efficient mechanism for providing easily assimilated biochemical energy.
The problem is over-consumption of refined sugars. This is a problem because the results of sugar metabolism (particularly fructose) are highly reactive (which is why they are so easy to metabolize), and can mess up your biochemistry by reacting with proteins and other macromolecules. The phytochemicals in plant sugar sources seem to reduce the biochemical side reactions of sugar metabolism which is probably why the plants synthesize many of them in the first place.
Refined sugars tend to spike the concentration of sugar in your body, leading to more possibility for reactions. Secondly, without the phytochemical stabilizers, the incidence of side-reactions is greater.
The body is a complex system, we need to stop trying to analyze it as if it were linear and reducible.
