The Bitter Lesson

Summary of The Bitter Lesson

by Ben Jaffe and Katie Malone

19m•March 15, 2026
Listen to Episode

Overview of Linear Digressions — "The Bitter Lesson"

This episode (hosts: Ben Jaffe and Katie Malone) explains Richard Sutton’s 2019 essay "The Bitter Lesson": across many AI domains, methods that exploit scale (more compute + more data + simple, general algorithms) repeatedly outperform approaches that rely on carefully hand-engineered domain knowledge. The episode traces historical examples (chess, language, vision), explains why builders feel anxiety as models rapidly improve, and gives practical guidance for what to build now to remain useful as models continue to scale.

Key historical examples illustrating The Bitter Lesson

  • Chess (1997 — Deep Blue)
    • Prior approach: hand-coded heuristics and strategic rules attempting to emulate human chess knowledge.
    • Deep Blue: won by massively improving search (brute-force evaluation of many positions). The simpler, scale-focused tactic beat elegant hand-designed heuristics.
  • Language (late 2000s — "The Unreasonable Effectiveness of Data")
    • Prior approach: sophisticated linguistic rules, grammars, and handcrafted datasets.
    • Google research (Peter Norvig et al.): simpler models trained on massive web-scale data outperformed complex, rule-heavy systems.
  • Vision (2012 — AlexNet)
    • Prior approach: hand-engineered visual features (edges, textures) based on human vision insights.
    • AlexNet: deep nets trained on raw pixels + large datasets (ImageNet) and scalable compute (GPUs) gave a dramatic performance leap — a phase shift not just incremental improvement.

The Bitter Lesson (core idea)

  • Distillation: Over decades, broad, general methods that leverage increasing computation and large datasets tend to outperform domain-specific, hand-crafted solutions.
  • Sutton’s warning: Hand-engineering can give short-term gains but is often overtaken by scale in the long run. Scale “learns” structure from data without needing costly human assumptions.
  • Analogy used in the episode: hand-engineering features is like writing an encyclopedia; scaling is like building a library that keeps adding books.

Practical guidance for AI builders (what to build vs. what to avoid)

  • Ask a decisive question for each feature: In a world with a near-perfect model, would this feature be:
    • Made unnecessary? (Likely throwaway/short-term work), or
    • Enhanced by a perfect model? (Likely durable investment)
  • Avoid heavy investment in solutions that patch current model weaknesses that scale will likely remove:
    • Example avoid: elaborate post-processing to fix LLM-generated JSON syntax, or complex context-compression hacks that only exist because context windows are currently limited.
  • Invest in complementary capabilities that scale won’t replace:
    • Retrievable, curated data that is not in public training sets (company-internal knowledge, proprietary datasets, time-sensitive info).
    • Robust retrieval/augmentation pipelines (RAG, long-term memory, vector stores) so models can access outside, up-to-date, or private context.
    • Integrations to systems and data sources that models cannot access by themselves (APIs, databases, enterprise systems).
    • Human-in-the-loop workflows where human judgment is genuinely load-bearing (safety-critical review, domain expertise, value judgments).
    • Instrumentation, evaluation, monitoring, and safety mechanisms (quality control and governance that persist as model capabilities change).

Main takeaways

  • Scale has repeatedly been the dominant force improving AI systems across domains.
  • Short-term engineering to patch current model limits is often temporary and may be obsoleted by future models.
  • Durable work is that which augments models (context, connections, human oversight) rather than competes with them.
  • It’s often hard to see which approach will win in the moment — the lesson is easier to see in hindsight — so prioritize work that remains valuable even if models become much stronger.

Notable quotes & metaphors

  • “The Bitter Lesson” — the recurring, uncomfortable lesson that scale wins over hand-crafted intelligence.
  • “Hand engineering features is kind of like writing an encyclopedia. Scaling is like building a library that keeps adding books.”
  • Practical test: “Would a perfect model make this feature unnecessary, or would it make it better?”

Quick action checklist for AI teams

  • Audit current features and classify: likely-obsolete vs. likely-durable under better models.
  • Shift resources away from brittle heuristics that fix current model bugs.
  • Invest in:
    • High-quality, proprietary data collection and indexing.
    • Reliable retrieval/augmentation and long-term memory systems.
    • System integrations (APIs, database access, automations).
    • Human-in-the-loop pipelines where human decisions are essential.
    • Continuous evaluation, monitoring, and safety tooling.
  • Keep learning and adapting: short-term experiments are fine, but expect to throw away some tactical work as models improve.

Closing note

The episode is both a caution and a permission slip: keep building and experimenting, but prioritize work that complements scalable models rather than competes with them. The landscape will keep changing, so design systems that leverage model improvements instead of being fully reliant on handcrafted fixes or brittle workarounds.