That's not at all clear!

> Language, when conveyed between conscious individuals creates a shared model of the world. This can lead to visualizations, associations, emotions, creation of new memories because the meaning is shared. This does not happen with mere syntactic manipulation. That was Searle's argument.

All of that is called into question with some LLM output. It's hard to understand how some of that could be produced without some emergency model of the world.