Doctors are inherently risk averse, all that means is “we don’t know, and there’s not much downside to not doing it, so we might as well advise against”.
Saying “no one can say” means they literally have no idea. It’s not all that different than the statement “no one can say that 5G doesn’t cause cancer”.
Note that phytoestrogens are categorically not human estrogen, they are simply compounds that mimic some of estrogen’s properties.
Also of note: breast milk contains estrogen.
