Count tokens used by GPT-4 and Llama for large texts (> 50k characters) (opens in new tab)

This web-app fixes the two main problems of OpenAI's tokenizer playground: (1) being capped at 50k characters, and (2) not supporting GPT-4/GPT-3.5 tokenizers.

Everything runs in-browser thanks to Transformers.js.