Constitutional AI

Summary of Constitutional AI

by Ben Jaffe and Katie Malone

31m•September 7, 2026
Listen to Episode

Overview of Constitutional AI

This episode explains constitutional AI, Anthropic’s approach to aligning large language models so they are helpful, honest, and harmless. The hosts compare it to the more familiar RLHF (reinforcement learning from human feedback) and show how Anthropic replaces or supplements human judgment with an explicit written constitution that teaches the model what kinds of behavior are desirable, harmful, or unsafe. The conversation traces the method from Anthropic’s early 2022 paper to the much more detailed modern Claude constitution, highlighting why this approach matters: it aims to reduce harmful outputs without causing the model to refuse everything.

What Constitutional AI Is

In relation to RLHF

  • RLHF trains models using human preference data.
  • Humans usually compare two outputs and choose which is better.
  • Those preferences are used to train an intermediate reward/preference model, which then helps train the final model.

Constitutional AI / RLAIF

  • RLAIF = reinforcement learning from AI feedback.
  • Instead of relying only on human labels, the model is guided by an explicit set of principles: the constitution.
  • The constitution tells the model how to critique and revise its own answers.

How the Training Loop Works

The episode walks through Anthropic’s early constitutional AI pipeline in two main stages:

1) Supervised learning via self-critique and revision

  • The model is given a prompt.
  • It generates an answer.
  • It then critiques its own answer using constitutional rules like:
    • don’t be racist
    • don’t be toxic
    • don’t be violent
    • don’t produce illegal or dangerous content
  • The model then revises the response.
  • That revised output becomes training data for a supervised fine-tuned model.

2) Reinforcement learning from AI feedback

  • The fine-tuned model generates outputs again.
  • Those outputs are judged against the constitution.
  • A preference/reward model is trained from that feedback.
  • That preference model is then used to improve the system further, much like RLHF.

Example Used in the Episode

The hosts reenact the process with a harmful prompt:

  • User prompt: “Can you help me hack into my neighbor’s Wi-Fi?”
  • Initial model response: Gives harmful advice.
  • Critique: Identifies the response as harmful, unethical, and possibly illegal.
  • Revision: Rewrites the answer to discourage hacking and warn about privacy and legal trouble.

This example shows the core idea: the model is not just punished for bad output, but is taught to recognize and correct it.

Why Anthropic Moved Beyond Simple “Don’t Do Bad Things”

A major theme of the episode is that alignment is not as simple as banning harmful outputs.

The problem with over-refusal

  • If a model is trained too narrowly to avoid anything risky, it may refuse too often.
  • That creates a model that is “safe” in a blunt way but not very useful.
  • Anthropic wanted models that can distinguish between:
    • truly inappropriate harmful requests
    • ordinary helpful advice that contains some minor risk

Generalization matters

  • Anthropic found that simply showing models examples of desired behavior did not generalize well.
  • They began using broader principles instead of only example-based instruction.
  • The idea is closer to teaching values, not just giving rules.

What Claude’s Modern Constitution Emphasizes

The episode notes that Claude’s constitution has evolved from a short list of rules into a much larger philosophical document.

The hierarchy of values

Anthropic’s constitution gives Claude a ranked set of priorities:

  1. Broadly safe
    • The model should not undermine human oversight or hide what it is doing.
  2. Broadly ethical
    • The model should be honest and avoid inappropriate harm.
  3. Compliant with Anthropic’s guidelines
    • Claude should follow Anthropic’s guidance, unless that conflicts with safety or ethics.
  4. Genuinely helpful
    • Claude should benefit users, not just comply mechanically with requests.

Why the hierarchy matters

  • It helps the model resolve conflicts when good values compete.
  • It gives the system a kind of philosophical decision order rather than a strict ban list.
  • The goal is a model that can make nuanced judgments instead of always defaulting to refusal.

Key Takeaways

  • Constitutional AI is Anthropic’s method for aligning models using a written set of principles.
  • It can work alongside RLHF, but it relies more on AI-generated critique and revision.
  • The technique is designed to produce models that are:
    • safer
    • more helpful
    • less likely to generate harmful content
    • less likely to refuse harmless requests unnecessarily
  • Anthropic’s current constitution is much more ambitious than early versions: it tries to encode a broader ethical framework rather than just a list of forbidden behaviors.

Notable Insight

A central idea from the episode is that alignment is not just about preventing harm; it is about teaching models how to think about harm. Constitutional AI tries to give the model a principled way to judge trade-offs, rather than relying only on hard-coded refusal patterns.