The fundamental difference is between the sugar ring (flexible, which each carbon having tetrahedral geometry, and localized electron density) and the aromatic benzene ring (flat trigonal geometry, with electrons delocalized over the ring).
If you attach chlorine to the latter, you end up with something that's very hard to break down (very persistent) and which will often (depending on exact structure) stick tightly to many biological molecules. Thus they can cause developmental disruption, odd forms of cancer, liver damage, etc. For example, TCDD (dioxin, a side product of many herbicide synthesis routes, a notorious component of Agent Orange formulations in Vietnam, etc.):
https://en.wikipedia.org/wiki/2,3,7,8-Tetrachlorodibenzodiox...
The chlorinated sugar ring is probably easier to break down but if this produces an activated chlorine species during metabolism, say in your liver, it might bind to who-knows-what and cause problems. Or it might just get excreted as a chlorine ion, no different from NaCl (table salt). I'd avoid it myself.
Some organo-chlorines (and -bromines) also occur in small amounts in many edible seaweeds, apparently often with the halide attached to fatty acids and lipids. Hence consuming large amounts of seaweed might not be the best idea either.
