Calculate the number of language model tokens for a string (opens in new tab)

(blog.mastykarz.nl)

17 pointswaldekm1y ago3 comments

3 comments

3 comments · 1 top-level

sva_1y ago· 2 in thread

  import tiktoken
  enc = tiktoken.encoding_for_model('o1-mini')
  len(enc.encode('some string'))

Does that work on anything other than openai?

No but most models have their own libs you can use

j / k navigate · click thread line to collapse