What is transformer?
A transformer is a neural network architecture using attention mechanisms to weigh relationships across sequences, powering modern language AI models.
What it means
The transformer is the neural network architecture behind modern generative AI. Older sequence models read sentences strictly word by word from left to right, often losing track of early context in long paragraphs and struggling to train in parallel on modern graphics hardware.
Transformers introduced self-attention: a mechanism that lets every token in a sequence weigh how strongly it relates to every other token at once, whether two words sit side by side or paragraphs apart.
Why it matters if you're a student
You do not need to memorise matrix equations unless you study machine learning, yet understanding why transformers scaled so fast explains today's industry landscape:
- Parallel training: transformers let labs train vastly larger models across clusters of chips simultaneously.
- Beyond text: the exact same sequence-attention idea now powers protein folding, audio, code and vision tools.
- Quadratic cost pressures: comparing every token against every other token gets computationally heavy as context windows grow huge.
An everyday example
Read the sentence: "The solicitor refused to sign the indemnity clause because it exposed the client." Self-attention is how the network connects "it" strongly to "clause" rather than "solicitor", tracking who did what across complex legal or technical prose.
Common misconception
People sometimes talk as if transformers solved reasoning permanently. Attention is brilliant at capturing statistical relationships across sequences, yet long multi-step arithmetic, strict logical constraints and novel out-of-distribution physics problems can still trip a pure transformer up without external verification tools.