I don't worry about side effects because in practice they don't cause me many problems (although I do focus on isolating mutable state). But if I thought about it a lot then I might start to worry about the theoretical possibilities. So I think the question is well phrased. "Worry" is mainly psychological.

Other advantages of pure FP might be thread safety but queues take care of most of this (and often locks are not hard to use). Or low-level compositionality. But as Alan Kay said, what we need are bigger objects. John Ousterhout seems to agree. Re-use in the small increases complexity.