After all, that's what Artificial General Intelligence would at least in part be about: finding and proving new math theorems, creating new poetry, making new scientific discoveries, etc.
There is even a new challenge that's been proposed: https://arcprize.org/blog/launch
> It makes sense that the process of thinking and the process of translating those thoughts into and out of language would be distinct
Yes, indeed. And LLMs seem to be very good at _simulating_ the translation of thought into language. They don't actually do it, at least not like humans do.