Tokenizer Arena

Every model sees text as tokens, and every model shreds the same text differently. Type below and watch five real tokenizers, the actual production vocabularies, split your text live in your browser. Fewer tokens means cheaper inference and more room in the context window. The differences are bigger than you think.

Why this matters: tokenization decides cost, speed, and fairness. A 200k vocabulary (GPT-4o) packs common English into single tokens, while a 2019-era 50k vocabulary (GPT-2) needs several. And languages like Malayalam can cost 10x more tokens per sentence than English, which means 10x the price and 10x less context for the same meaning. Try the presets above to see it happen.
Tokenizers run entirely client-side via Transformers.js, using each model's real vocabulary files. Nothing you type leaves this page. Related: my Rust benchmark of tokenizer implementations, tokenizer-arena.