Count tokens with tiktoken: Python examples
tiktoken is OpenAI's open-source BPE tokenizer, and the quickest way to count tokens exactly for OpenAI models in code. This page gives tested Python examples and shows how to check your results against our browser token counter. More token guides are in tokens and context windows.
By the Chelure Enterprises Editorial Team. Last reviewed . How we check our sources.
Install and count
pip install tiktoken
import tiktoken
enc = tiktoken.get_encoding("o200k_base")
ids = enc.encode("Hello world")
print(len(ids), ids) # 2 [13225, 2375]
print([enc.decode([i]) for i in ids]) # ['Hello', ' world']
print(enc.decode(ids)) # Hello worldThese outputs are from a run of tiktoken 0.13.0 on November 13, 2026. The same text under cl100k_base gives [9906, 1917], also two tokens.
Pick the encoding for a model
enc = tiktoken.encoding_for_model("gpt-4o")
print(enc.name) # o200k_basetiktoken's README associates cl100k_base with GPT-4 and uses o200k_base in its main example; in our installed version encoding_for_model maps gpt-5, gpt-4.1, gpt-4o and the o-series to o200k_base, and gpt-4 and gpt-3.5-turbo to cl100k_base. For newer OpenAI models, confirm which encoding applies.
| Encoding | Models it maps to in tiktoken 0.13.0 (encoding_for_model) |
|---|---|
| o200k_base | gpt-5, gpt-4.1, gpt-4o, o1, o3, o4-mini (and fine-tunes of gpt-4o) |
| cl100k_base | gpt-4, gpt-3.5-turbo, text-embedding-3-small and large, text-embedding-ada-002 |
A reusable counting function
def count_tokens(text, encoding="o200k_base"):
enc = tiktoken.get_encoding(encoding)
return len(enc.encode(text, disallowed_special=()))
print(count_tokens("<|endoftext|> counted as plain text")) # 11Without disallowed_special=(), tiktoken raises a ValueError when the text contains a special-token string such as <|endoftext|>. Counting that text as ordinary characters, as above, is what you usually want for a size estimate.
From tokens to cost
Multiply the token count by the model's input price per million tokens, divided by 1,000,000. For example, a 10,500-token prompt at $2 per million input tokens costs $0.021. The prompt cost calculator does this for any model, and the cost calculator handles volume, caching and batch discounts.
Check against the browser counter
Our token counter reproduces tiktoken in the browser. We compared it with tiktoken on more than 1,600 strings in both encodings and the token ids matched every time, so a count from either should agree. The counter shows real token boundaries, which is useful for seeing why a string costs what it does.
Frequently asked questions
How do I count tokens with tiktoken?
Install it with pip install tiktoken, get an encoding with tiktoken.get_encoding("o200k_base") or tiktoken.encoding_for_model("gpt-4o"), and take the length of encoding.encode(text).
Which encoding should I use?
tiktoken's README associates cl100k_base with GPT-4 and uses o200k_base in its main example; encoding_for_model maps a model name such as gpt-4o to its encoding. For newer models, confirm the encoding with OpenAI.
Why does tiktoken raise an error on text like <|endoftext|>?
By default it refuses to encode special-token strings found in your text. Pass disallowed_special=() to count them as ordinary text, which is what our browser counter does.
Does tiktoken work offline?
It downloads each encoding's vocabulary file the first time you use it, then caches it. After that it works offline. Our browser counter loads the same public vocabularies as static files.
Does tiktoken count Claude or Gemini tokens?
No. It implements OpenAI's encodings. Use Anthropic's and Google's own token-counting endpoints for those models.
Sources
Primary sources opened on November 13, 2026. Prices come from our verified price table (see the methodology).