I think people who say this are imagining LLMs work something like a statistical model. Maybe it's doing a linear regression or works like a Markov chain. It's not.

A single artificial neuron sort of works like that. But that's sort of like saying a single transistor is just an electronically controlled switch, so the only thing computers can do is switching. It's true in some sense that computers are just doing a lot of switching, but it turns out all this switching is Turing-complete. That means computers can theoretically compute anything that's possible to compute given enough time and memory, which includes anything a human could figure out.

Similar principle applies to LLMs. Using probabilities is part of what they do, but that doesn't preclude them from using logic and rules of inference.