The Transformer model explained
Almost every large language model you can call through an API descends from one 2017 paper, "Attention Is All You Need". This guide explains what the Transformer is, what attention does, and why the architecture shapes cost and context limits. Start with what is a large language model if you want the wider picture; section hub: AI models explained.
Prices last verified against official provider pages on . How we verify prices. Written by the Chelure Enterprises Editorial Team.
What the paper proposed
Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser and Polosukhin proposed the Transformer, a network architecture "based solely on attention mechanisms, dispensing with recurrence and convolutions entirely". Before it, the leading sequence models processed text step by step. The Transformer processes all positions of a sequence together and lets every position attend to every other. On WMT 2014 it reached 28.4 BLEU for English-to-German, over 2 BLEU above the previous best results including ensembles, and 41.8 BLEU for English-to-French as a single model, after 3.5 days of training on eight GPUs.
The parts
| Component | What it does |
|---|---|
| Token embeddings | Turn each token (see what is a token) into a vector of numbers |
| Self-attention | For each position, computes how much to draw on every other position, using learned projections called queries, keys and values |
| Multi-head attention | Runs several attention computations in parallel so the model can track different relationships at once |
| Feed-forward layers | Apply a further transformation to each position after attention |
| Positional information | Gives the model the order of tokens, since attention alone does not encode it |
| Stacked layers | Repeat attention and feed-forward blocks many times, so later layers work on richer representations |
Why this shapes cost and limits
The paper's complexity comparison lists self-attention as costing operations proportional to the square of the sequence length per layer, with every position connected to every other in a constant number of steps. That all-to-all connectivity is what lets a model use context from anywhere in a prompt, and it is one reason context windows have limits and long prompts are expensive to serve. Generation then proceeds one token at a time, which is part of why output tokens cost more. Our context window table shows how far current models have pushed the limit.
From the Transformer to today's LLMs
Later work scaled the architecture. Kaplan et al. (2020) found that language-model loss follows a power law in model size, dataset size and compute, and Brown et al. (2020) showed a 175 billion parameter model performing tasks from a few examples in the prompt. Instruction tuning and human feedback then turned such models into assistants: see the LLM guide for that history.
Frequently asked questions
What is a Transformer?
A Transformer is a neural network architecture introduced by Vaswani et al. in 2017 that relies solely on attention mechanisms, dispensing with recurrence and convolutions. It underlies today's large language models.
What is attention?
Attention lets the model weigh how much each token in the input should influence the representation of every other token, so context from anywhere in the text can be used directly instead of being passed along step by step.
Why do long prompts get expensive?
In the original paper's complexity comparison, self-attention cost per layer grows with the square of the sequence length, which is one reason long contexts are costly to serve. Providers' pricing pages do not break down their costs, so treat this as background, not as an explanation of any provider's prices.
What did the original Transformer achieve?
The paper reports 28.4 BLEU on WMT 2014 English-to-German translation, improving on the previous best results by over 2 BLEU, and 41.8 BLEU on English-to-French after 3.5 days of training on eight GPUs.
Are all LLMs Transformers?
The large language models behind today's chat and API products are based on the Transformer, although architectures keep evolving and providers do not always publish details of their models.
Sources
- Vaswani et al.: Attention Is All You Need (arXiv:1706.03762)
- Kaplan et al.: Scaling Laws for Neural Language Models (arXiv:2001.08361)
- Brown et al.: Language Models are Few-Shot Learners (arXiv:2005.14165)
Primary sources opened on November 13, 2026. Prices come from our verified price table (see the methodology).