Transformer Architecture — Core Concepts

The Paper That Broke Everything Open

In June 2017, eight Google Brain researchers published a paper with a bold title: “Attention Is All You Need.” It introduced the Transformer architecture. Within two years, it had obliterated every prior approach to natural language processing. Within five, it was running inside every major AI product on the planet.

Most people get this wrong: Transformers aren’t just a better language model. They’re a general-purpose architecture for understanding sequences — which turns out to describe almost everything interesting: text, images, audio, video, protein structures, even code.

What Was Wrong Before

To understand why Transformers were revolutionary, you need to know what they replaced.

Recurrent Neural Networks (RNNs) processed sequences step by step. Word 1 → Word 2 → Word 3. Each step fed into the next. The problem: by step 50, information from step 1 had been diluted through 49 transformations. Long-range dependencies (like knowing what “it” refers to 20 words earlier) were nearly impossible.

LSTMs and GRUs added “memory” to RNNs and helped, but they still processed sequentially. This meant you couldn’t parallelize training across a GPU’s thousands of cores — the network had to wait for step n before starting step n+1. Training was slow. Scaling was painful.

Transformers fixed both problems at once.

The Core Idea: Self-Attention

The key innovation is self-attention (also called scaled dot-product attention).

Here’s the intuition: for every word in a sentence, the model asks — “Which other words in this sentence are relevant to understanding this word?” Then it computes a weighted average of all words’ representations, where words that are more relevant get more weight.

Take the sentence: “The animal didn’t cross the street because it was too tired.”

When processing “it,” self-attention lets the model look at every other word simultaneously. It assigns high attention weight to “animal” (the referent) and lower weights to “street” and “cross.” The resulting representation of “it” is now informed by “animal” — the model has resolved the ambiguity without any explicit grammar rule.

The Three Vectors: Queries, Keys, and Values

Self-attention works through three learned vectors for each word:

Vector	What it represents
Query (Q)	What this word is looking for
Key (K)	What this word has to offer
Value (V)	The actual content to retrieve

The process: multiply each Query against all Keys to get attention scores, apply softmax to normalize them into weights, then take a weighted sum of all Values.

It’s like a fuzzy search engine. “Street” queries for location-related keys. “Animal” queries for subject-related keys. Each word’s final representation is a blend of information from all other words, weighted by relevance.

Multi-Head Attention: Looking for Multiple Things at Once

One attention head can only look for one type of relationship at a time. Multi-head attention runs 8, 16, or more attention heads in parallel — each one free to specialize.

In practice, different heads learn different things. One head might track subject-verb relationships. Another might track coreferents (which “it” refers to). Another might capture positional relationships. Nobody programs these roles — the model discovers them during training.

Positional Encoding: Injecting Order

Here’s a subtle problem: if you look at all words simultaneously, you lose track of word order. “Dog bites man” and “Man bites dog” have the same words — but different meanings.

Transformers solve this by adding positional encodings — mathematical signals injected into each word’s embedding that encode its position in the sequence. The original paper used sine and cosine waves at different frequencies. Newer models use learned positional embeddings or more sophisticated approaches like RoPE (Rotary Position Embedding).

The Encoder-Decoder Structure

The original Transformer had two parts:

Encoder: Takes the input (e.g., a French sentence) and builds a rich representation of it. Each layer applies self-attention + a feedforward network. Layers stack on top of each other — typically 6-24 layers deep.

Decoder: Generates the output (e.g., the English translation) one token at a time. It attends to both its own previous outputs (masked self-attention) and the encoder’s representation (cross-attention).

Modern models often use just one half. GPT uses decoder-only. BERT uses encoder-only. Both are Transformers.

Why It Scaled So Well

The reason Transformers took over is not just accuracy — it’s that they scale beautifully.

Unlike RNNs, all tokens can be processed in parallel. A GPU with 10,000 cores can work on all 1,000 words of a document simultaneously. This made it economically viable to train on billions of tokens, then trillions.

The empirical finding that shocked researchers: scaling just works. Bigger model + more data + more compute → reliably better performance. No one fully understands why, but the results are undeniable. GPT-2 had 1.5 billion parameters. GPT-4 is estimated at over 1 trillion.

Common Misconception: “Transformers Understand Language”

They don’t — not in the way humans do. Transformers are extraordinarily good at learning statistical patterns in sequences. They compress an enormous amount of structure about language into their weights. But there’s ongoing debate about whether that constitutes “understanding” or very sophisticated pattern matching.

What’s not debatable: they produce outputs that often appear remarkably intelligent, and they’ve demonstrated capabilities nobody predicted — like learning to write code, solve math problems, and reason through arguments — just from next-token prediction on text.

One Thing to Remember

A Transformer processes every word in a sentence simultaneously and uses self-attention to figure out which words should inform the meaning of each other word — no step-by-step reading, no forgetting the beginning. That parallelism is why it could be trained at massive scale, and why it powers every major AI product today.