What is a Token (in AI)? — AI Glossary

In AI, a token is the basic unit of text that a language model reads and generates. Tokens are not the same as words — they’re pieces of words (or sometimes whole words, punctuation marks, or even single characters) that the model uses to break down text into manageable chunks it can process mathematically. Understanding tokens is essential because they directly affect how AI models work, what they can remember, and how much they cost to use.

As a rough rule of thumb: 1 token ≈ 0.75 words in English. So 100 words is approximately 133 tokens. The sentence “The quick brown fox” is 4 words but roughly 4-5 tokens.

How Tokens Work

Language models don’t read text the way humans do — they can’t process raw characters or whole words directly. Instead, they use a tokenizer to split text into tokens before processing. The tokenizer is trained to create an efficient vocabulary — one that represents common words as single tokens but breaks rare or long words into pieces.

Here’s how different text gets tokenized:

• “cat” → 1 token (“cat”)
• “running” → 1 token (“running”)
• “unbelievable” → 3 tokens (“un”, “believ”, “able”)
• “ChatGPT” → 3 tokens (“Chat”, “G”, “PT”)
• “Hello!” → 2 tokens (“Hello”, “!”)

The most common tokenization algorithm used today is Byte Pair Encoding (BPE). It starts with individual characters and progressively merges the most common pairs until it reaches a target vocabulary size (typically 30,000–100,000 tokens). This means common words become single tokens while rare words get split into their component pieces.

Interestingly, the same word tokenizes differently across languages. English is token-efficient; other languages (especially those with different scripts or complex morphology) often require more tokens per word. This means GPT-4 processing French or Chinese text uses more tokens — and therefore costs more — than equivalent English text.

Why Tokens Matter

Tokens matter for three practical reasons:

1. Context window limits. Every language model can only process a fixed number of tokens at once — called its context window. GPT-4’s standard context window is 128,000 tokens (roughly 96,000 words or a 400-page book). Claude’s context window goes up to 200,000 tokens. Once you exceed this limit, the model can no longer “remember” earlier parts of the conversation.

2. API pricing. Commercial AI APIs like OpenAI and Anthropic charge per token — typically per 1,000 or 1,000,000 tokens. As of early 2025, GPT-4o charges approximately $2.50 per million input tokens and $10 per million output tokens. If you’re building an application that processes thousands of documents, token count directly determines your costs.

3. Model performance. Models have a fixed “attention budget” — their ability to relate tokens to each other decreases as context gets longer. Keeping prompts focused and concise often produces better results than padding them with unnecessary text.

Tokens in Practice

Here are practical implications of understanding tokens:

• Estimating costs: Use OpenAI’s tokenizer tool to count tokens in any text before processing it through an API.
• Optimizing prompts: Shorter, denser prompts use fewer tokens and often work better. Remove filler words and redundant instructions.
• Chunking documents: For documents that exceed context limits, use RAG to split them into chunks and retrieve only relevant pieces.
• Code and structured data: JSON, XML, and code are often token-inefficient because they have lots of repetitive structural characters. Minify or compress data formats when passing to models.

Common Misconceptions About Tokens

Tokens are not words. This surprises many people. The mismatch means that counting words is an unreliable way to estimate token usage. Always use a tokenizer to get accurate counts.

More tokens don’t always mean better performance. Adding more context can actually hurt performance if the relevant information gets “diluted” in a long window. Research from Stanford has shown that models tend to focus most on the beginning and end of long contexts — a phenomenon called “lost in the middle.”

Token limits apply to input AND output combined. The context window includes everything — your prompt, the conversation history, and the model’s generated response. A 128k context window doesn’t mean you can send 128k tokens of input and still get a long response.

For more technical detail, see the tokenization overview at Grokipedia, the BPE tokenization paper at arXiv, or use OpenAI’s interactive tokenizer.

Key Takeaways

• In one sentence: A token is the basic text unit AI models process — roughly 0.75 words on average — and it determines both what a model can remember and how much API usage costs.
• Why it matters: Token limits define what an AI can process in one request; token pricing defines what it costs to build AI applications.
• Real example: GPT-4o’s 128,000-token context window can hold roughly a 400-page book worth of text in a single conversation.
• Related terms: Context Window, LLM, Transformer, RAG

Frequently Asked Questions

How many tokens is 1,000 words?

Approximately 1,333 tokens in English. The ratio is roughly 1 word = 1.33 tokens on average. For more precise estimates, paste your text into the OpenAI tokenizer.

How much does 1 million tokens cost?

It varies by model and provider. As of early 2025: GPT-4o costs about $2.50 per million input tokens; Claude 3.5 Sonnet costs about $3 per million input tokens; GPT-4o mini costs about $0.15 per million input tokens. Prices are falling rapidly — they’ve dropped roughly 10x in the past two years.

What is the token limit for free ChatGPT?

Free ChatGPT (GPT-3.5) has a context window of about 16,385 tokens. ChatGPT Plus uses GPT-4o, which has a 128,000-token context window. The free tier may limit how much of that window you can use per session.

Do images also use tokens?

Yes. Multimodal models like GPT-4o process images by converting them into tokens too — the number depends on image resolution. A standard 512×512 image might use 170-500 tokens. High-resolution images cost significantly more tokens (and money) to process.

Can I reduce token usage to lower API costs?

Yes. Effective strategies include: write concise system prompts (remove all unnecessary words), use RAG to only pass relevant document chunks instead of entire documents, cache repeated context, use smaller/cheaper models for simpler tasks, and compress structured data formats before sending them to the model.

What are tokens in ChatGPT?

In ChatGPT and other LLMs, a token is the basic unit of text the model reads and generates — roughly 3–4 characters or about 0.75 words on average. The phrase ‘Hello, world!’ is about 4 tokens. Models like GPT-4 have a context window measured in tokens (e.g., 128,000 tokens ≈ about 96,000 words), and API pricing is billed per token for both input and output.

Why do AI tools count tokens?

Tokens matter for two reasons: cost and memory. API pricing is billed per thousand tokens, so longer prompts and longer conversations cost more. Context windows are also measured in tokens — once you exceed the limit, the model can’t ‘see’ earlier parts of the conversation, which is why AI can seem to forget things in very long chats. Keeping prompts concise reduces cost and keeps important context within the window.

Get free AI tips delivered daily → Subscribe to Beginners in AI

You May Also Like

• What Is Artificial Intelligence
• AI Glossary
• AI for Dummies
• How Does AI Work
• Essential AI Skills