Skip to content
Better HN
Top
Best
Ask
Show
New
Jobs
Search
⌘K
0 points
wongarsu
3y ago
0 comments
Save
Share
Tokens. Short or common words tend to be one token, while less common words are composed of multiple tokens. For GPT OpenAI gives the rule of thumb that on average you need four tokens to encode three words, and LLaMA should be similar
0 comments
1 comments · 1 top-level
top
newest
oldest
worldsayshi
3y ago
Well that's for sure bigger than my context size.
j
/
k
navigate · click thread line to collapse