undefined | Better HN

0 pointsctoth2y ago0 comments

This is (probably) wrong, see:

https://gwern.net/scaling-hypothesis And specifically the part where it discusses addition under the heading Blessings of Scale.

0 comments

3 comments · 1 top-level

haltist2y ago· 2 in thread

So why can't GPT do arithmetic? At what scale do you suppose it will manage to add numbers?

famouswaffles2y ago

GPT-4 can do arithmetic just fine. In fact, despite being severely hobbled by Tokenization (https://arxiv.org/abs/2310.02989), it can perform arithmetic better than you could without a dedicated tool or pad.

Der_Einzige2y ago

Besides your answer on tokenization being right on the mark (gwern even has articles discussing this exact issue - https://gwern.net/gpt-3#bpes), that linked paper is extremely powerful for theoretical thinking about this problem. I literally just left a comment claiming that almost all of human thought is encoded in LLMs, leading almost all tasks to be considered interpolation.

If this is true, I suppose based on what we know of the curse of dimensionality it makes sense. Very thought-provoking work.

1 more reply

j / k navigate · click thread line to collapse