What it means

Language models do not read whole words or raw letters the way people do. Before processing your prompt, a tokeniser chops text into standard chunks called tokens and maps each chunk to a number. Common English words usually take one token, while rarer technical terms, code identifiers or non-English words split into several smaller sub-word tokens.

Models read tokens in, calculate probabilities one token at a time, and bill API usage based on how many input and output tokens pass through the system.

Why it matters if you're a student

Tokens explain three practical quirks you will notice straight away when building or studying with AI:

→ Cost and speed: verbose outputs cost more and take longer because every single output token is generated sequentially → Context budgets: a hundred-page PDF burns through token limits fast, especially if full of tables or raw HTML → Character-level blind spots: because models see multi-letter chunks rather than individual letters, naive prompts can trip over letter counting or reversing strings

An everyday example

The word "student" might sit in a model's vocabulary as one single token, whereas "decarbonisation" or an unfamiliar chemical name gets split into three or four pieces. Code indentation, brackets and numbers also consume tokens alongside ordinary prose.

Common misconception

Students sometimes equate one token with one English word. As a rough rule of thumb in English prose, three-quarters of a word per token is closer, while dense source code, JSON files or languages with different scripts can use noticeably more tokens for the exact same meaning.

  • A context window is the maximum amount of text, code or files an AI model can hold and process at one time across both your input and its response.

  • A large language model (LLM) is an AI system trained on text to predict tokens in sequence, letting it draft, summarise, translate and write code.

  • A transformer is a neural network architecture using attention mechanisms to weigh relationships across sequences, powering modern language AI models.

  • Natural language processing (NLP) is the branch of AI enabling computers to parse, search, classify, translate and generate written or spoken language.

Where it matters

Explore how this concept shows up inside real graduate workflows and sector briefings:

Articles that mention it

No published issue names this term directly yet. Browse our latest industry articles or read our practical AI guides.

Get the fortnightly briefing

theVANTAGEd examines AI across real industries with critical thinking, every two weeks. Written by students.