2D Tokenization for Large Language Models (opens in new tab)

This article is about how we process text before passing it to large language models, the problems with this approach, and an alternative solution.