What is Token Throughput?

What it is: Token throughput measures how many tokens an AI system can produce per second — a combined measure of how fast individual answers feel and how many users a system can serve at once.
Who it’s for: Anyone comparing AI providers, building agents, or trying to understand “TPS” numbers in a benchmark
Best if: You’ve seen “120 tokens/sec” next to a model name and wondered if that’s good
Skip if: You already optimise vLLM and TensorRT-LLM for a living

What is Token Throughput?

Token throughput is the speed at which a language model produces output, measured in tokens per second (TPS). A token is roughly a word or part of a word — “unbelievable” is three tokens, “the” is one. If a model runs at 100 tokens per second, it generates about 75 English words every second. The metric is the most direct gauge of how fast an AI feels to use and how much work an AI service can do for the price of its hardware.

There are actually two related numbers people call “throughput.” Per-user throughput is how fast one streaming response feels — the words spilling onto your screen. Aggregate throughput is the total tokens the whole system produces across all simultaneous users. A provider can be fast on one and slow on the other, and they often optimize for whichever matters most for their workload.

Think of it like a restaurant. Per-user throughput is how quickly one diner gets their courses. Aggregate throughput is how many diners the kitchen serves per hour. A fine-dining restaurant optimises the first; a stadium concession stand optimises the second.

Why It Matters

Throughput is the user-facing side of inference economics. Two providers running the same open-weights model can have wildly different throughput because of how they batch, cache, and parallelize. Groq and Cerebras have built whole companies on serving the same Llama and Qwen models at 5–10x the throughput of standard GPU clouds. For a chatbot, throughput governs whether the response feels alive or sluggish. For an agent loop that needs to make 30 model calls before it answers, throughput governs whether the workflow takes 5 seconds or 5 minutes.

Throughput is also tied to cost. Cloud providers charge by token consumed, not by time, so a faster system also tends to be cheaper per request because the same hardware serves more users. This is why infrastructure investments in memory cache, custom chips, and better batching kernels keep paying off.

How It Works

Throughput comes from three multiplied factors: how fast a single token-generation step is on the chip, how much can be produced in parallel through batching, and how well the system avoids redundant work via caching. A frontier 70B-parameter model on a single H100 GPU might run at 30 tokens per second per user. The same model with 64 users batched together might run at 1,500 aggregate tokens per second — each user perceives a slight per-user slowdown, but the system as a whole is far more efficient.

Specialised inference chips like Groq’s LPU or Cerebras’s WSE break the trade-off by holding the entire model in extremely fast on-chip SRAM, eliminating the memory bandwidth bottleneck that limits GPU throughput. The downside is they cost more per chip and only support certain model sizes — but where they fit, they can quote 800–2,000 tokens per second on real-world models.

Examples

Chat experience: Claude Sonnet at ~80 tokens per second feels conversational — text streams faster than you can read it. The same model at 20 tokens per second feels slow.

Agent loops: A coding agent that needs 30 model calls to solve a task is bottlenecked by throughput. Higher TPS turns a 5-minute wait into a 30-second wait.

Voice and real-time: Voice agents need throughput high enough that the model finishes speaking before the user gets impatient. Below ~150 tokens per second the lag becomes audible.

Sources

• NVIDIA — TensorRT-LLM inference performance overview
• vLLM — throughput & batching documentation
• Artificial Analysis — provider throughput benchmarks

Last reviewed: May 2026

You May Also Like

• What is a Token (in AI)?
• What is Inference?
• What is Memory Cache?
• What is Parallelization?
• AI Glossary: 100+ Terms Every Beginner Needs to Know