AI Inference 101

Summary of AI Inference 101

by Ben Jaffe and Katie Malone

27m•October 26, 2026
Listen to Episode

Overview of AI Inference 101

This episode kicks off a new season of Linear Digressions focused on AI inference—the process of taking a fully trained, frozen model and using it to generate outputs for users. The hosts explain why inference is a major engineering challenge that often gets less attention than model training, and they introduce the core performance tradeoffs that determine how fast and efficiently LLMs respond.

The central idea: inference has two main phases—prefill and decode—and each is constrained by different bottlenecks. Understanding those bottlenecks is key to making AI systems faster, cheaper, and more scalable.

What Inference Is

Training vs. inference

  • Training is when a model learns its weights.
  • Inference is when those weights are frozen and the model is used to generate tokens for a prompt.
  • Most public-facing AI use happens here: chatbots, summarization, code generation, and more.

Why inference matters

  • It’s the part of AI that users directly experience.
  • It involves real-world deployment problems: latency, throughput, memory limits, and hardware efficiency.
  • A lot of the “magic” of AI happens in inference, not just in training.

The Two Main Phases of Inference

1. Prefill: time to first token

  • This is the initial stage where the model processes the entire input prompt.
  • It is a large parallel computation.
  • The main goal here is reducing latency—how long you wait before the first output appears.

2. Decode: token generation

  • After prefill, the model generates output one token at a time.
  • This phase is more serial and often slower in a different way.
  • The main metric here is tokens per second.

How Attention Drives the Cost of Inference

The transformer’s core mechanism

  • LLMs use transformer architectures.
  • The most computationally important part is attention.

Query, key, value

Each token has:

  • a query: what it’s looking for,
  • a key: what it represents,
  • a value: the content it contributes.

The model compares queries against keys to decide what to attend to, then uses the values to update token representations.

A helpful analogy

The episode compares attention to attending a conference:

  • Your query is what you want to learn.
  • Talk titles are the keys.
  • The content of the talk is the value.
  • After attending selected talks, your understanding changes, just like a token’s representation changes after attention.

Why Prefill and Decode Bottleneck Differently

Prefill is compute-bound

  • During prefill, the model processes many tokens at once.
  • This involves lots of matrix multiplication and heavy computation.
  • Bigger compute resources help here.

Decode is memory-bound

  • During decode, the model repeatedly reloads its weights to produce each new token.
  • The limiting factor is often memory bandwidth, not raw compute.
  • Faster memory access can improve output speed more than simply adding more processing power.

Hardware and Model Size Matter

Bigger model, more memory pressure

  • Large models require moving more weights around.
  • Sometimes the model is too large to fit on a single chip, requiring distribution across machines.

Smaller models are faster to serve

  • Smaller models need less data movement.
  • That makes them attractive for low-latency inference and local deployment.

Better hardware helps—but only up to a point

  • More powerful chips can improve performance.
  • But if you are memory-limited, extra compute alone won’t help much.
  • Real-world performance depends on balancing compute capacity and memory bandwidth.

The Big Picture: Optimizing for Balance

The episode frames inference as an engineering problem of finding the sweet spot where:

  • the processor is fully utilized,
  • memory bandwidth keeps up,
  • neither compute nor memory is sitting idle.

This balance is what allows AI systems to serve users efficiently at scale.

Key Takeaways

  • Inference is the runtime use of a trained model, not the training process itself.
  • It has two major phases:
    • Prefill = large, parallel, compute-heavy
    • Decode = serial, memory-heavy
  • Attention is the main driver of transformer cost.
  • The system’s speed depends on whether it is compute-bound or memory-bound.
  • Prompt length, model size, and hardware choice all affect inference performance.
  • Transformer attention is fundamentally O(n²), which creates a major scalability challenge.

What’s Next

The episode ends with a cliffhanger: if transformer attention scales so expensively, how do modern AI systems make inference tractable at all?

The next installment will explore the engineering tricks that help avoid the worst effects of the n-squared attention problem and make large-scale inference practical.