Humans come in all levels of intelligence. Yes there might be a human somewhere that believes that they should eat rocks or add glue to pizza. But as a generality we can assume that adult "humans" know not to eat rocks.

I think it’s a mistake to equate the language ability of these models with general intelligence, even if the language produced is excellent.

I think some people see AGI that way (some nebulous unfalsifiable thing), but that’s not what I’m arguing. I think there’s a strong case to be made that at a minimum, AGI’s core “knowledge” will need to be made up of far more than just a lossy textual representation of the world.