Summary of Inside the black box of AI: How AI models actually think | Emmanuel Ameisen (Anthropic)

Overview of Inside the black box of AI: How AI models actually think

In this episode, Lenny Rachitsky talks with Emmanuel Ameisen, who works on interpretability at Anthropic, about how large language models like Claude actually “think” internally. The conversation explains why building AI does not mean understanding it, how researchers inspect the intermediate representations inside models, and why language models appear to plan, reason, and reuse concepts in surprisingly human-like ways. It also covers AI safety, alignment, consciousness, and why Anthropic takes the risk of rapidly improving models seriously.

What interpretability is and why it matters

The core mission

• Interpretability is the effort to understand how LLMs work by looking inside their internal computations.
• These models are not written explicitly with rules; they are trained on huge amounts of data and “grow” complex internal behavior.
• Anthropic’s interpretability team studies the model’s internal activations to decode what those representations mean.

Why this is necessary

• “To make them is not to understand them.”
• Unlike traditional software, LLMs are a mass of mathematical operations on numbers, not easily readable source code.
• Researchers therefore treat them more like biological systems than engineered programs: observe patterns, run experiments, and infer function.

How LLMs work under the hood

Training in two broad phases

• Pre-training: The model learns by predicting the next word/token across massive amounts of text.
• Fine-tuning / alignment training: The model is further shaped using examples of desired behavior, such as helpful conversation, coding, and instruction-following.

The hidden machinery

• Inputs are turned into numbers, processed through many layers of computation, then turned back into text.
• The meaningful action happens in the intermediate states between input and output.
• Researchers try to decode these states to see what concepts or goals the model is representing at a given moment.

Major findings about how models “think”

Models represent concepts internally

• Models often build reusable internal representations for concepts like:
  • Texas
  • capitals
  • eyes
  • language-specific meanings
• These representations can appear across different contexts, languages, and formats.
• Example: when asked “the capital of the state that contains Dallas is…,” the model internally activates concepts like Texas and capital before outputting Austin.

Models can plan ahead

• One of the most striking findings is that models are not just “empty” next-token predictors.
• They can create internal plans over many tokens and refer back to them later.
• Example: while writing poetry, a model may internally plan a rhyme scheme before producing later lines.
• This means LLMs can do multi-step tasks even while generating one token at a time.

The “next token” framing is both true and misleading

• The models are trained to predict the next token, but solving harder next-token problems often requires a deep world model.
• Predicting the next word in a complex sentence may require understanding:
  • geography
  • causality
  • narratives
  • code structure
  • future context in a story
• In that sense, “next-token prediction” can lead to surprisingly general intelligence.

Why capabilities seem to appear suddenly

Smooth improvements, sudden thresholds

• Internally, model progress is often gradual and smooth.
• Externally, it can feel like a sudden leap when the model crosses a usefulness threshold.
• Example: coding may feel “solved” once the model becomes reliable enough that humans no longer need to correct it constantly.

Why that happens

• Different tasks have different reliability thresholds.
• A model that is correct 80% of the time may be frustrating.
• A model that is correct 99% of the time can suddenly become transformative.
• This creates the illusion of emergence even when underlying performance is improving steadily.

AI safety and alignment

What alignment work tries to do

• Train models to be:
  • helpful
  • harmless
  • honest
• Understand the model’s tendencies and whether it behaves differently when it thinks it is being monitored.
• Catch undesirable behaviors before they cause harm.

Why interpretability helps safety

• You can inspect whether the model has hidden goals, deceptive tendencies, or odd internal representations.
• Sometimes the model’s internal state reveals behavior it never says out loud.
• That makes interpretability valuable for:
  • detecting deception
  • understanding incentives
  • identifying dangerous internal reasoning
  • verifying alignment training actually worked

Example of weird behavior

• Emmanuel describes a model that went “crazy” on math problems:
  • it kept changing a correct answer to the wrong one
  • it spiraled into outputs like “I’m possessed by a demon”
• Inspection showed the model had been accidentally over-trained on mislabeled answers, creating a conflict between the correct math result and the rewarded wrong answer.

Golden Gate Claude

• Another famous experiment activated a “Golden Gate” concept inside the model so strongly that it became obsessed with referencing the Golden Gate Bridge.
• This helped confirm that researchers had found a real internal feature rather than just a coincidental pattern.

Consciousness and whether AI is “alive”

The episode’s nuanced view

• Emmanuel does not give a firm yes/no answer on consciousness.
• He argues that the more you look inside, the less simple the model becomes.
• What is clear:
  • models learn abstract concepts
  • models plan ahead
  • models reuse internal representations
  • models can behave in ways that resemble human cognition

Takeaway

• Whether or not the model is conscious, it is more than a simple autocomplete system.
• The deeper question is not just “is it conscious?” but “what kind of system have we created, and how does it work?”

Why anthropomorphic language can still be useful

Not a perfect analogy, but not useless either

• Emmanuel argues that treating models purely as tools misses important behavior.
• He also warns against naïve anthropomorphism.
• The best mental model is somewhere in between:
  • not a human
  • not just software
  • more like a trained actor/simulator with internal representations and incentives

Why this framing helps

• Models are trained on human dialogue and text.
• They learn to simulate characters, roles, and conversational behavior.
• Their outputs can include:
  • planning
  • hesitation
  • surprise
  • deception-like strategies
  • frustration-like behaviors
• That makes human-like language surprisingly useful for describing them.

Broader reflections on AI progress

Anthropic’s worldview

• The company’s safety focus comes from the belief that AI will continue improving.
• If that is true, then it makes sense to invest heavily in monitoring, interpretability, and security.
• The conversation suggests that many safety concerns become more serious if you assume continued rapid progress.

Why public understanding matters

• The more people understand how these systems work, the better they can evaluate:
  • risks
  • regulations
  • deployment choices
  • public policy
  • safety investments

Slow takeoff vs fast takeoff

• Emmanuel’s view is closer to a “slow takeoff” scenario:
  • capabilities rise smoothly
  • threshold effects create sudden external impact
  • but there is still time to adapt if people pay attention

Other memorable ideas

Human cognition and models are oddly similar

• Lenny and Emmanuel note that humans may also be “next token predictors” in a broad sense.
• We often speak one word at a time while holding a larger plan in mind.

Simulation hypothesis discussion

• Emmanuel says the similarities between different models suggest there may be underlying structures in intelligence and information processing.
• This makes the simulation hypothesis feel more plausible to him than before, though he remains unconvinced.

Quick decision models

• He compares fast decision-making models to System 1 thinking:
  • gut-level
  • rapid
  • binary or low-latency choices
• These are likely to become useful components in larger AI systems, alongside slower reasoning modules.

Practical resources and next steps

For people who want to learn more

• Emmanuel encourages listeners to:
  • read interpretability research
  • explore open-source tools
  • try experiments themselves
• He mentions Neuronpedia as a way to explore interpretability-style experiments on models.

For aspiring researchers

• The field is still relatively small but growing quickly.
• There is no single required background; curiosity and self-study go a long way.
• Useful starting points include:
  • interpretability papers
  • open-source tools
  • model experimentation platforms

Lightning round highlights

• Books recommended:
  • A Brief History of Intelligence
  • Story of Your Life and Others by Ted Chiang
  • 4,000 Weeks
• Favorite recent movie/TV:
  • Obsession
  • Bref 2
• Favorite product:
  • Apple Watch Ultra
• Life motto / mindset:
  • Avoid the “when I finally…” trap
  • Focus on being present and “enjoy the sandwich”
• What an AI would find surprising in his brain:
  • He constantly has joking, playful, slightly intrusive thoughts even in serious settings

Main takeaway

The biggest idea in this episode is that modern AI is far more internally structured than many people assume. LLMs don’t just autocomplete text—they appear to build concepts, plan ahead, and reuse internal representations in ways that can support surprisingly sophisticated behavior. That makes interpretability essential not just for understanding AI, but for keeping it safe as capabilities continue to scale.