Computers can reason and do symbolic manipulation too (e.x.
https://en.wikipedia.org/wiki/Computer_algebra).
There’s no reason to believe AGI will be composed of only language models. I suspect it will be a combination of multiple models and a lot of other software glued together in clever ways.