What is Memory Cache (in AI)?

What it is: Memory cache (often called prompt caching or KV caching) is an AI optimization that stores the work the model already did on repeated parts of a prompt, so it doesn’t have to redo it on every request.
Who it’s for: Anyone using AI APIs, building agents, or wondering why long-context calls suddenly got cheap
Best if: You’ve seen “cached input” line items on an OpenAI or Anthropic invoice
Skip if: You only use AI through a chat interface and never look at API pricing

What is Memory Cache (in AI)?

Memory cache is the technique AI providers use to avoid re-computing the same intermediate work twice. Every time a large language model processes text, it generates a giant pile of internal numbers called the key-value cache (or KV cache) — essentially the model’s “notes” about what it has read so far. If the next request starts with the same text, the model can reuse those notes instead of generating them from scratch. The result is dramatically faster and cheaper inference for any prompt that begins with shared context.

An analogy: imagine a librarian who has just finished reading a 600-page legal contract for you and is now answering your questions about it. If you come back tomorrow with a new question, you don’t want her to start over from page 1 — she should remember everything she read yesterday. Memory cache is the system that lets her do that. The shared context (the contract) is cached; only the new question incurs the full reading cost.

Anthropic markets this feature as prompt caching. OpenAI calls it cached input. Google calls it context caching. They all describe the same underlying idea: don’t redo work you’ve already done.

Why It Matters

Memory cache is the single biggest reason long-context AI is affordable. A 200,000-token context window would be ruinously expensive if every request paid the full processing cost. With caching, the system pays the heavy cost once when the cache is built, then every subsequent request inside the cache window is up to 10x cheaper and several times faster.

This changes what’s economically feasible. AI agents that re-read a 50,000-token system prompt on every step. Customer-support bots grounded on a multi-megabyte FAQ. Coding agents that load an entire repository once and then ask hundreds of follow-up questions. None of these workloads were practical before caching matured in 2024–2025.

How It Works

Inside a transformer, every token in the input gets converted into a pair of vectors called a key and a value. Computing those K/V pairs across millions of tokens is what makes long prompts expensive. The cache simply saves the K/V pairs in fast memory and replays them on the next request that starts with the same prefix.

The catch is that caches only match on identical prefixes. If you change a single character at the start of the prompt, the cache misses and the work is redone. Good API design therefore puts stable content (system prompts, documents, tool definitions) at the front and variable content (the user’s new question) at the end. Cache lifetimes are short — Anthropic’s default is 5 minutes, with 1-hour as an option — because keeping K/V pairs in GPU memory is itself expensive.

Examples

Coding agent on a large repo: Claude Code reads 50,000 tokens of repository context once, caches it, then handles 30 follow-up edits in the same session — only the new instructions are billed at full price.

Customer-support bot: A retailer attaches a 100,000-token product manual to every conversation. With caching, the manual is processed once per cache window; thousands of customer questions reuse it at a fraction of the cost.

Document Q&A: A user uploads a 600-page legal brief and asks 20 questions in a row. Each question pays only for itself plus a small cache-read fee — not the full cost of re-reading the brief.

Sources

• Anthropic — Prompt caching documentation
• OpenAI — Prompt caching guide
• Google — Gemini context caching

Last reviewed: May 2026

You May Also Like

• What is a Context Window?
• What is a Token (in AI)?
• What is Inference?
• What is RAG?
• AI Glossary: 100+ Terms Every Beginner Needs to Know