> but we actually know it's a program and how it works. There is no mystery.

You're right, we do know how it works. Your mistake is concluding that because we know how LLMs work and they're not that complicated, but we don't know how the brain works and it seems pretty complicated, therefore the brain can't be doing what LLMs do. That just doesn't follow.

You made exactly the same argument in the opposite direction, asking if my rubric for "can reason with intelligence and have an opinion" is "seems like it", and your rubric for "thinking is not a token predictor driven by matrix multiplications" is "seems like it".

You can make a case for the plausibility of each conclusion, but that's doesn't make it a fact, which is how you're presenting it.

Then please go ahead and explain how something can solve novel logical puzzles (i.e. ones that are not present in its training set) without some capacity for reasoning. You're claiming that it is "generating texts that looks like ..." - so what is the "..." in this case? I posit that the word that should be placed there is solution, and then you need to explain why that is not ipso facto a demonstration of the ability to reason.