Home › Tokens and context windows › What is a token in AI

What is a token in AI? Tokenization explained with examples

Language models do not read letters or words; they read tokens. This guide explains what a token is, how byte-pair encoding builds the vocabulary, and shows real token splits and cross-language counts we measured with OpenAI's tokenizer. To count your own text, use the token counter; the full topic is in tokens and context windows.

Quick answerA token is the unit of text a language model reads and writes: a word, part of a word, a few digits or a symbol. OpenAI's rule of thumb is about 4 characters or 0.75 words per token in English. In our sample, a Russian sentence needed 27 tokens on o200k_base and 57 on cl100k_base, against 18 for English.

By the Chelure Enterprises Editorial Team. Last reviewed 13 November 2026. How we check our sources.

What a token is

OpenAI defines tokens as commonly occurring sequences of characters, and gives a rough rule of thumb that one token is about 4 characters or 0.75 words of English text. A token can be a whole word (" world"), a piece of a word ("token" + "ization"), a few digits ("123"), a symbol, or a space joined to the word that follows it. The model never sees the raw characters, only the integer id of each token.

Real token splits

These splits were produced with OpenAI's o200k_base encoding (a visible-space mark shows a leading space). The last column shows the older cl100k_base count for comparison.

Text	o200k_base tokens	Count	cl100k_base count
Hello world	Hello \| ␣world	2	2
hello	␣hello	1	1
tokenization	token \| ization	2	2
Straßenbahnhaltestelle	Stra \| ßen \| bah \| nh \| al \| test \| elle	7	7
ChatGPT	Chat \| GPT	2	3
2026	202 \| 6	2	2
1234567890	123 \| 456 \| 789 \| 0	4	4
https://createchz.com	https \| :// \| create \| ch \| z \| .com	6	6
def f(x): return x**2	def \| ␣f \| (x \| ): \| ␣return \| ␣x \| ** \| 2	8	8
😀	😀	1	2

Notice that a common word with a leading space is one token, a long German compound splits into seven pieces, a four-digit year splits into 202 + 6, numbers are cut into groups of up to three digits, and an emoji can be a single token. You can reproduce any of these in the counter, which shows the real boundaries.

How the vocabulary is built: byte-pair encoding

Byte-pair encoding starts from single symbols and repeatedly merges the most frequent adjacent pair into a new symbol, until the vocabulary reaches the target size. Sennrich, Haddow and Birch (2015) showed that encoding rare and unknown words as sequences of subword units let translation models handle an open vocabulary. OpenAI's tiktoken library applies BPE to bytes, so any text, in any script, can be represented. The encodings are public: cl100k_base is associated with GPT-4 and o200k_base with GPT-4o and later models.

The same sentence in eight languages

We translated one sentence ("The European Union has adopted new rules for artificial intelligence that apply to companies around the world") and counted tokens. This is a single sample, not a benchmark, and the translations are ours, but it shows the direction.

Language	Characters	o200k_base tokens	vs English	cl100k_base tokens	vs English
English	110	18	1.00×	18	1.00×
German	125	24	1.33×	31	1.72×
French	129	25	1.39×	33	1.83×
Spanish	116	22	1.22×	26	1.44×
Russian	121	27	1.50×	57	3.17×
Arabic	91	22	1.22×	63	3.50×
Chinese	21	14	0.78×	24	1.33×
Japanese	39	30	1.67×	43	2.39×

Under cl100k_base the Russian and Arabic sentences needed about three times the tokens of English; o200k_base cut that to about 1.5 times and 1.2 times. Chinese needed fewer tokens than English in this sample because each character carries a lot of meaning, while Japanese needed more. For budgeting, this means an API bill for German, Russian or Japanese text can run noticeably higher than the same content in English.

Tokens across providers

Each provider has its own tokenizer, so counts differ. Anthropic states that its current tokenizer produces more tokens for the same text than its earlier models (1M tokens is about 555K words on the current tokenizer against about 750K before), which is why we label Claude and Gemini counts as estimates. Prices per token are in the price comparison.

Frequently asked questions

What is a token in AI?

A token is the unit of text a language model reads and writes: a whole short word, part of a longer word, a number chunk, a punctuation mark or a space plus a word. OpenAI says tokens represent commonly occurring sequences of characters.

How many characters or words is a token?

OpenAI gives a rough rule of thumb of about 4 characters or 0.75 words per token for English text. Our measurements on one sentence show how far that can drift for other languages and for code.

Why do other languages use more tokens?

Tokenizer vocabularies are built from training text where English dominates, so common English words are often one token while the same meaning in another script can take several. In our sample, o200k_base needed 27 tokens for a Russian sentence that took 18 in English, and cl100k_base needed 57.

What is byte-pair encoding?

Byte-pair encoding (BPE) builds a vocabulary by repeatedly merging the most frequent pairs of symbols. Its use for subword units in neural machine translation was introduced by Sennrich, Haddow and Birch (2015); OpenAI's tiktoken applies BPE at the byte level.

Are tokens the same across models?

No. Each provider uses its own tokenizer, so the same text gives different counts. Anthropic notes that its current tokenizer produces more tokens for the same text than earlier Claude models, and our counter is exact only for OpenAI's encodings.

Why does token count matter?

API prices, context windows and output limits are all measured in tokens, so your token count sets your bill and decides whether a prompt fits.

Sources

Primary sources opened on November 13, 2026. Prices come from our verified price table (see the methodology).

Token splits and counts were measured with tiktoken on November 13, 2026; the script is build/_measure_tokens.py in our build.