KV Cache Fundamentals

Summary of KV Cache Fundamentals

by Ben Jaffe and Katie Malone

24m•November 2, 2026
Listen to Episode

Overview of KV Cache Fundamentals

This episode explains why the KV cache (key-value cache) is one of the most important engineering optimizations behind modern LLM inference. Ben Jaffe and Katie Malone walk through how attention works, why it becomes expensive as prompts grow, and how caching previously computed key/value tensors makes multi-turn and long-context LLM usage practical at scale.

What the KV Cache Is

  • In transformer attention, each token produces a key (K) and value (V) representation.
  • These key/value pairs do not change after they’re computed for a given context.
  • Instead of recalculating them on every new token, systems store them in memory and reuse them.
  • This stored memory is the KV cache.

Why it matters

  • Without caching, attention would repeatedly recompute over the full growing context.
  • That creates an O(n²) cost pattern as the conversation gets longer.
  • Reusing the KV cache makes inference much more tractable, bringing the effective cost closer to O(n) behavior with memory tradeoffs.

KV Cache vs. Context Window

  • Context window: the model’s maximum token capacity; a fixed property of the model.
  • KV cache: the actual memory used during a live request or conversation.
  • You can think of the context window as the ceiling, and the KV cache as the current working memory being consumed in that session.

How It Works During Inference

The episode breaks inference into two main phases:

1. Prefill

  • The model processes the initial prompt.
  • It computes K and V for all prompt tokens.
  • Those values are stored in the cache.

2. Decode

  • The model generates tokens one at a time.
  • Each new token’s K/V values are appended to the cache.
  • The cache grows as generation continues.

Different Usage Regimes

The hosts describe several common LLM workloads and how they stress the KV cache differently:

Short chatbot exchanges

  • Small number of turns.
  • Usually far below the context limit.
  • Cache usage stays modest.

Long conversations

  • Multi-minute or multi-hour chats.
  • The full conversation history must remain in cache.
  • Memory usage grows steadily and can become substantial.

Large single-shot tasks

  • Examples: summarizing a long document or a RAG-style query.
  • Heavy prefill cost up front.
  • Cache may only be reused a little before being discarded.

Coding agents

  • Often the most demanding case.
  • Long-running sessions plus tool calls, sub-agents, and large context history.
  • These can rapidly inflate cache usage and system cost.

The Core Memory Problem

  • KV cache space is finite and must be reserved from available GPU memory.
  • If you reserve too much, memory is wasted.
  • If you reserve too little, the system may need to evict and recompute cache entries, which hurts performance.
  • Earlier systems could waste 20–40% of memory, and in worse cases even 60–80% due to poor allocation strategies.

Paged Attention: The Key Optimization

The episode highlights a major breakthrough from Berkeley in 2023: paged attention.

Main idea

  • Instead of reserving one large contiguous cache block upfront, allocate cache in small chunks as needed.
  • This is similar to how operating systems use paging for virtual memory.

Benefits

  • Avoids both over-allocation and under-allocation.
  • Dynamically grows the cache during generation.
  • Can reduce memory waste to under 4%.
  • Improves throughput by 2x to 4x at the same latency.

Why it matters in practice

  • Paged attention became foundational to vLLM, one of the most widely used inference engines.
  • It’s now a core part of modern open-source LLM serving infrastructure.

Practical Implications for Users

The episode also connects KV cache mechanics to real products like Claude Code:

  • Cached input tokens are cheaper than uncached (“cold”) tokens.
  • Long-idle sessions may lose their warm cache and need recomputation.
  • Tools may suggest clearing context to avoid expensive cache rebuilds.
  • Cache hit rates can often be inspected through usage commands.
  • Caches are model-specific: switching models usually forces a new cache build.
  • Frequent model switching is inefficient; staying on one model preserves cache reuse.

Main Takeaways

  • The KV cache is a central reason LLM inference is feasible at scale.
  • It turns repeated attention computation into reusable memory.
  • Memory management is now one of the key frontiers of LLM serving.
  • Paged attention is the major modern solution for efficient KV cache allocation.
  • Real-world AI products actively encourage cache reuse because it materially lowers cost and improves performance.

Bottom Line

The episode’s core message is that modern LLM serving is not just about model quality—it’s about inference engineering. The KV cache, and especially paged attention, are crucial optimizations that make long-context, multi-turn, high-throughput LLM applications practical and affordable.