And if you ask an LLM about an effective cure for cancer and it spits out to drink bleach, are you going to follow it?
When you ask 'who knows' that's the point of research which was my original comment here. The same goes true for some random asshole telling you to drink bleach as it does an LLM, except people seemingly have hyped themselves into believing the LLM is more right somehow instead of being trained on every random asshole ever.