What is token?
A token is the basic chunk of text, code or data an AI model processes, often equal to a word, part of a word, a number or a punctuation character.
What it means
Language models do not read whole words or raw letters the way people do. Before processing your prompt, a tokeniser chops text into standard chunks called tokens and maps each chunk to a number. Common English words usually take one token, while rarer technical terms, code identifiers or non-English words split into several smaller sub-word tokens.
Models read tokens in, calculate probabilities one token at a time, and bill API usage based on how many input and output tokens pass through the system.
Why it matters if you're a student
Tokens explain three practical quirks you will notice straight away when building or studying with AI:
→ Cost and speed: verbose outputs cost more and take longer because every single output token is generated sequentially → Context budgets: a hundred-page PDF burns through token limits fast, especially if full of tables or raw HTML → Character-level blind spots: because models see multi-letter chunks rather than individual letters, naive prompts can trip over letter counting or reversing strings
An everyday example
The word "student" might sit in a model's vocabulary as one single token, whereas "decarbonisation" or an unfamiliar chemical name gets split into three or four pieces. Code indentation, brackets and numbers also consume tokens alongside ordinary prose.
Common misconception
Students sometimes equate one token with one English word. As a rough rule of thumb in English prose, three-quarters of a word per token is closer, while dense source code, JSON files or languages with different scripts can use noticeably more tokens for the exact same meaning.