Tiktoken-rs: Batteries-included Rust library for working with GPT-4's tokenizer (opens in new tab)

(github.com)

2 pointszurawiki3y ago1 comments

1 comments

In case others find this useful. Just updated for GPT-4, tiktoken-rs is a Rust crate you can use to tokenize text sent to and from OpenAI's LLM models

j / k navigate · click thread line to collapse

Tiktoken-rs: Batteries-included Rust library for working with GPT-4's tokenizer (opens in new tab)

(github.com)

2 pointszurawiki3y ago1 comments

1 comments

zurawikiOP3y ago

In case others find this useful. Just updated for GPT-4, tiktoken-rs is a Rust crate you can use to tokenize text sent to and from OpenAI's LLM models

j / k navigate · click thread line to collapse