Tokens in other languages: why German, Russian and Hindi cost more
AI is priced per token, and the same content needs a different number of tokens in different languages. We measured two sentences in 15 languages with OpenAI's exact tokenizers to show how far the cost moves. Measure your own text in the token counter; the basics are in what is a token in AI, and the section hub is tokens and context windows.
By the Chelure Enterprises Editorial Team. Last reviewed . How we check our sources.
The measurement
We used two sentences ("The European Union has adopted new rules for artificial intelligence that apply to companies around the world. Please send the invoice by email before the end of the month.") translated into each language by us, and counted tokens with OpenAI's o200k_base and cl100k_base encodings. The multiplier is tokens in that language divided by English tokens.
| Language | Characters | o200k_base tokens | vs English | cl100k_base tokens | vs English |
|---|---|---|---|---|---|
| English | 172 | 31 | 1.00x | 31 | 1.00x |
| German | 187 | 37 | 1.19x | 48 | 1.55x |
| French | 190 | 39 | 1.26x | 49 | 1.58x |
| Spanish | 188 | 39 | 1.26x | 43 | 1.39x |
| Italian | 188 | 46 | 1.48x | 55 | 1.77x |
| Portuguese | 168 | 38 | 1.23x | 47 | 1.52x |
| Dutch | 182 | 37 | 1.19x | 48 | 1.55x |
| Polish | 183 | 47 | 1.52x | 62 | 2.00x |
| Turkish | 161 | 40 | 1.29x | 60 | 1.94x |
| Russian | 186 | 43 | 1.39x | 89 | 2.87x |
| Arabic | 150 | 35 | 1.13x | 102 | 3.29x |
| Hindi | 154 | 47 | 1.52x | 153 | 4.94x |
| Chinese | 37 | 25 | 0.81x | 39 | 1.26x |
| Japanese | 60 | 44 | 1.42x | 64 | 2.06x |
| Korean | 69 | 36 | 1.16x | 73 | 2.35x |
What the numbers show
- Latin-script European languages (German, French, Spanish, Italian, Portuguese, Dutch) needed roughly 1.2 to 1.5 times the English tokens on o200k_base.
- Non-Latin scripts cost the most on the older tokenizer. On cl100k_base, Hindi needed 4.9 times and Arabic 3.3 times the English tokens; o200k_base cut Hindi to 1.5 times and Arabic to 1.1 times.
- Chinese used fewer tokens than English (25 against 31) on o200k_base, while Japanese and Korean used more.
What it means for budgets
If you process Hindi, Russian or Arabic text with a model that uses the older cl100k_base tokenizer, expect to pay two to five times the token cost of the same content in English; newer tokenizers narrow the gap but do not close it. Context windows are also counted in tokens, so a 1M-token window holds less text in those languages; see context window sizes. Other providers use different tokenizers, so confirm with their token-counting tools.
Budget checklist
- Count a representative sample of your own text, not a sentence.
- Price it with the prompt cost calculator.
- Prefer a model with a newer tokenizer for non-Latin-script work, and re-check when you switch models.
Frequently asked questions
Why do some languages use more tokens?
Tokenizer vocabularies are learned from training text where some languages are far more common than others, so common English words are often a single token while the same meaning in another script can take several. In our sample, the same two sentences took 31 tokens in English, 43 in Russian and 47 in Hindi on o200k_base; on the older cl100k_base Hindi took 153.
Does a newer tokenizer help?
Yes in our sample. OpenAI's o200k_base needed far fewer tokens than cl100k_base for non-Latin scripts: Hindi 47 versus 153, Russian 43 versus 89, Arabic 35 versus 102.
Is Chinese cheaper in tokens?
In this sample Chinese used fewer tokens than English (25 against 31 on o200k_base), because each character carries more meaning. Japanese and Korean used more.
Does this affect cost and context?
Yes. API prices and context windows are counted in tokens, so the same content in a language that tokenizes less efficiently costs more and fills the context window sooner.
Are these numbers a benchmark?
No. Two short sentences per language, with translations by us, show the direction, not the size, of the effect for your content. Measure your own text with the token counter. Other providers' tokenizers will give different counts.
Sources
- OpenAI: tiktoken (encodings o200k_base and cl100k_base)
- OpenAI: API concepts (tokens)
- Sennrich, Haddow and Birch: subword units for NMT (arXiv:1508.07909)
Primary sources opened on November 13, 2026. Prices come from our verified price table (see the methodology).
Counts measured with tiktoken on November 13, 2026 (build/_measure_languages.py). Translations are ours and illustrative.