undefined | Better HN

0 pointsisaacfung2y ago0 comments

The current gen llms tokenize numbers digit by digit unlike earlier llms.

0 comments

3 comments · 2 top-level

mike_hearn2y ago· 1 in thread

No, it doesn't help. Bloomberg tried this and it didn't seem to make much difference.

If someone else is interested in the Bloomberg tokenizer:

They don't. Which you can easily check with any of the dozen web apps currently implementing the GPT-4o tokenizer.

j / k navigate · click thread line to collapse