GPU Architecture for LLM Inference: Memory, Bandwidth and Hardware Explained

Summary of GPU Architecture for LLM Inference: Memory, Bandwidth and Hardware Explained

by Ben Jaffe and Katie Malone

19m•November 9, 2026
Listen to Episode

Overview of GPU Architecture for LLM Inference

Ben Jaffe and Katie Malone explain how GPUs power modern LLM inference, why inference has two very different phases, and how GPU memory hierarchy, bandwidth, and interconnects shape real-world performance. The episode also compares consumer, data center, and next-gen AI hardware, and closes with a look at specialized inference chips and a newer architecture pattern called disaggregated inference.

Core Idea: Why GPUs Matter for LLMs

  • GPUs are the workhorse of AI inference in data centers.
  • They were originally designed for graphics and parallel computation, not AI.
  • Their strength is doing many simple operations at once, which maps well to:
    • graphics rendering
    • matrix math in deep learning
    • large-scale inference workloads

How LLM Inference Actually Works

The episode breaks inference into two distinct stages:

1. Prefill

  • Processes the entire prompt in parallel.
  • Builds internal representations and the KV cache.
  • Is compute-bound:
    • benefits from high FLOPS
    • wants a powerful processor

2. Decode

  • Generates output token by token.
  • Is memory-bound:
    • depends heavily on fast data movement
    • needs efficient memory access and bandwidth

Main takeaway: one chip rarely excels equally at both, so inference systems must balance compute and memory carefully.

GPU Memory Hierarchy and Bandwidth

The episode explains the memory stack from fastest/smallest to slowest/largest:

On-chip SRAM

  • Extremely fast
  • Very limited capacity
  • Example figures discussed:
    • ~20 TB/s bandwidth
    • ~50 MB capacity

High Bandwidth Memory (HBM)

  • Off-chip but still on the board
  • Much larger, but slower than SRAM
  • Example figures discussed:
    • ~3–4 TB/s bandwidth on an NVIDIA H100
    • ~80 GB capacity

NVLink

  • Connects GPUs on the same board
  • Lets one GPU access another GPU’s resources quickly
  • Faster than external networking, but slower than local memory

InfiniBand

  • Data-center network for moving data between boards
  • Much slower than on-board connections, but enables scaling across many machines

Hardware Tiers Discussed

Consumer / Laptop-Class Hardware

  • Example: MacBook Air
  • Roughly:
    • 8 GB memory
    • ~250 GB/s bandwidth
  • Good for small models only
  • Most LLMs will not fit comfortably

Prosumer / Small Local AI Machines

  • Example: NVIDIA DGX Spark
  • Roughly:
    • 128 GB memory
    • ~273 GB/s bandwidth
  • Can run much larger local models
  • The speaker suggests it can handle models up to around 200B parameters under the right conditions

Data Center Workhorse

  • Example: NVIDIA H100
  • Roughly:
    • 80 GB HBM
    • strong board-level networking
    • often deployed in 8-GPU systems
  • Designed for high-throughput production inference

Next-Gen Systems

  • Example: NVIDIA Vera Rubin
  • Described as a much larger, more expensive system-level platform
  • Roughly:
    • 200–300 GB HBM
    • very high aggregate bandwidth, around 20 TB/s
  • Optimized for demanding production inference at scale

Beyond NVIDIA: Other Chips in the Inference Market

The episode notes that NVIDIA dominates, but other companies are building inference-specific hardware:

  • Groq
    • Optimized for low latency and more predictable performance
    • Especially strong on prefill
    • Mentioned as having a narrower gap between P50 and P99 latency
  • Cerebras
    • Specialized AI/inference hardware
  • Google TPU
    • Tensor Processing Unit for AI workloads
  • AWS chips
    • Trainium / Trainium 2 for training
    • Inferentia 2 for inference

Why Specialized Inference Hardware Matters

  • Training and inference have different demands.
  • Inference benefits from a tighter balance of:
    • compute
    • memory capacity
    • memory bandwidth
  • That’s why companies are designing chips and systems specifically for inference rather than using one general-purpose solution for everything.

Disaggregated Inference: Splitting Prefill and Decode

One of the episode’s most interesting ideas is disaggregated inference:

  • Prefill runs on one machine
    • compute-heavy
    • uses the largest processors available
  • The resulting KV cache is sent over fast networking
  • Decode runs on a different machine
    • optimized for memory access and token generation

Why this helps

  • Lets each stage use hardware tailored to its own bottlenecks
  • Improves utilization of expensive AI infrastructure
  • Adds complexity, but can boost performance significantly

Key Takeaways

  • Prefill = compute-heavy; decode = memory-heavy.
  • GPU performance for LLMs is mostly about balancing FLOPS, memory capacity, and bandwidth.
  • Local SRAM is fast but tiny; HBM is larger but slower.
  • NVLink and InfiniBand help scale inference across GPUs and racks.
  • NVIDIA dominates the market, but Groq, Cerebras, Google, and AWS are all building specialized alternatives.
  • Disaggregated inference is an emerging optimization that separates prompt processing from token generation.

Practical Implication

If you’re choosing hardware for LLM inference, the “best” machine depends on the workload:

  • Small local models: consumer or prosumer hardware
  • Production chatbots: multi-GPU data center systems
  • Ultra-optimized scale: specialized inference chips and disaggregated architectures