Overview of Invisible LLM Failures and AI Fluency with Chris Potts (Stanford)
This Linear Digressions episode features an interview with Stanford linguistics professor Chris Potts about how people interact with AI chatbots, why LLMs can fail in ways users don’t notice, and what it means to become a more “fluent” AI user. The conversation moves from Potts’s path from linguistics and swearing research into AI, to the limits of human intuition about LLMs, to practical questions about how these systems should be built and used.
Chris Potts’s Path From Linguistics to AI
From swearing to corpora to NLP
Potts explains that his route into AI was indirect:
- He originally studied swearing in language and why society treats it specially.
- That led him to corpus analysis, where he could study real language use at scale.
- From there, he began using NLP tools like parsers and POS taggers.
- Over time, that evolved into a research program focused on AI and human-language interaction.
Why this moment matters
He argues that LLMs are historically unusual:
- They are non-human systems that use language fluently.
- They are creating behaviors that even their builders do not fully predict.
- For linguists, they provide an existence proof that some complex language abilities can emerge from data-driven learning, not just innate mechanisms.
Invisible Failure Modes in Chatbot Interactions
What “invisible failure” means
Potts defines an invisible failure as a case where:
- Something went wrong in the interaction,
- But the user gave no obvious signal of frustration or correction,
- So the failure is hidden in the transcript or only becomes clear later.
In other words, the model may have been wrong, and the user may have left with incorrect information, but no explicit error was recorded.
Examples discussed
The interview highlights several common patterns:
- Death spiral: the user keeps trying to get the system to answer correctly, but the conversation never resolves.
- Silent contradiction: the AI says one thing early and the opposite later, and no one appears to notice.
- Silent mismatch: the model answers a slightly different question than the one asked, and the user seems to accept it.
Why this matters
Potts’s point is that these failures are not just edge cases:
- They appear frequently in real conversations.
- They may be invisible at the time even if they look obvious in hindsight.
- They can quietly produce bad outcomes, especially when users trust the model too much.
AI Fluency: The Difference Between Expert and Novice Users
What fluent users do differently
Potts describes fluent AI users as people who take an augmentative stance:
- They collaborate with the model instead of just delegating.
- They refine goals iteratively.
- They catch mistakes early.
- They nudge the model back on track.
How novice users behave
Less fluent users tend to be more delegative:
- They give a prompt and expect a great answer.
- They assume the system is highly reliable or near-superhuman.
- They are less likely to course-correct or challenge the output.
Why fluency matters
The episode suggests fluency is not just a style preference:
- It may be a causal factor in better outcomes.
- It could be teachable and therefore widely beneficial.
- Increasing fluency across users could reduce invisible failures and improve overall AI use.
Potts’s Research Methods and Annotation Challenges
Why labeling AI conversations is hard
The interview goes into the practical difficulty of analyzing chatbot transcripts:
- Some interactions are too complex for straightforward human annotation.
- Models are not deterministic, so even the same prompt can yield different labels.
- Different frontier models can disagree in systematic ways.
The workaround
Potts and his collaborators:
- First tried human-style annotation,
- Then realized the task was too hard to do reliably by hand,
- So they used LLMs as a team of annotators, with human guidance.
Key takeaway
This led to a broader point:
- LLMs don’t simply replace humans in annotation.
- They introduce their own quirks and variability.
- Research methods have to adapt to the fact that these systems are neither perfectly reliable nor fully human-like.
Do Invisible Failures Go Away as Models Improve?
The short answer: partially
Potts says the systems are improving:
- Overall failure rates are going down.
- But the shape of the failure distribution looks strikingly similar.
Why that matters
This suggests:
- Better models still fail in familiar ways.
- Improvements reduce frequency, not necessarily the underlying pattern.
- Users still need to be alert, especially in high-stakes or subtle tasks.
Design Implications: Build for Friction, Not Just Convenience
Why friction can be useful
A central theme of the discussion is that good AI systems may need intentional friction:
- Product design usually tries to remove friction.
- But in AI, friction can encourage users to think critically.
- It can push users toward verification rather than blind acceptance.
The tradeoff
Potts notes a major tension:
- Users often prefer systems that sound confident, validating, and smooth.
- But those qualities can correlate poorly with correctness.
- Companies may optimize for what users like in the moment, even if that hurts long-term reliability.
Where this is already working
He points to software development as a strong use case because:
- It has relatively high user fluency.
- It is more verifiable than many other domains.
- Errors are easier to catch when code can be run and checked.
For less verifiable domains—like design, theory, or writing—the burden on the user to assess correctness becomes much higher.
What Potts Is Most Interested in Now
Current research directions
Potts says his lab is focused on several frontiers:
- Interpretability: understanding how simple mechanisms produce complex behavior.
- Control and model improvement: adjusting behavior in existing models and influencing future architectures.
- Tokenizer-free models: working directly at the character/byte level, especially for multilingual settings.
- Language model programming: building more structured ways to direct LLM behavior.
- Skill files and customization: studying whether these tools actually improve productivity or create better interaction patterns.
Big open questions
He is especially interested in questions like:
- Do these tools make users more productive in practice?
- Do they create useful friction or harmful friction?
- How can we measure their real impact on workflow quality?
Main Takeaways
- LLMs are best understood not as magic or as human minds, but as alien tools with unusual behaviors.
- Many chatbot errors are invisible, meaning users may not notice them at the time.
- AI fluency matters: skilled users collaborate with models, verify outputs, and iterate.
- Better model performance does not eliminate the need for critical human oversight.
- The future of useful AI may depend on designing systems that encourage thoughtful engagement, not just smooth satisfaction.
Notable Ideas and Phrases
- “Invisible failure modes” — errors that happen without explicit user awareness.
- “Augmentative stance” — treating AI as a collaborator rather than a delegate.
- “The alien creature” metaphor — LLMs are language users, but fundamentally unlike humans.
- “Design with friction in mind” — a product principle meant to improve reliability and critical thinking.
Closing Note
The episode blends linguistics, AI research, and product design into a thoughtful argument: as LLMs become more capable, the biggest challenge may not be making them feel more human, but helping humans use them more intelligently.