Community
GPT-4o tokenizer splits "strawberry" into a single token, explained in animated video
The video walks through o200k_base tokenization, showing that " strawberry" is one token (id 101830) and how byte pair encoding builds word pieces. It also cites DeepSeek-R1 and DeepSeek-V3 letter-counting behavior from OpenAI's 2025 paper.