undefined | Better HN

0 pointssiwatanejo1mo ago0 comments

> All 7 books come to ~1.75M tokens

How do you know? Each word is one token?

0 comments

You can download the books and run them through a tokenizer. I did that half a year ago and got ~2M.

j / k navigate · click thread line to collapse

0 pointssiwatanejo1mo ago0 comments

> All 7 books come to ~1.75M tokens

How do you know? Each word is one token?

You can download the books and run them through a tokenizer. I did that half a year ago and got ~2M.

j / k navigate · click thread line to collapse