undefined | Better HN

0 pointsmrob1mo ago0 comments

Character counting errors are a side effect of tokenization, which is a performance optimization. If we scaled the hardware big enough we could train on raw bytes and avoid it.

0 comments

3 comments · 1 top-level

teiferer1mo ago· 2 in thread

No, tokenization is not the only reason. A next-word predictor has fundamentally a hard time executing algorithms, even as simple as counting.

mitthrowaway21mo ago

Counting is one of the algorithms that can be expressed by a RASP program, which transformers closely approximate.

MarkusQ1mo ago

Close famously counts in horseshoes and hand grenades. Algorithms, just as famously, are a domain where off-by-one is still wrong.

j / k navigate · click thread line to collapse