Overview of Customer support is the gnarliest AI problem
This Stack Overflow Podcast episode features Ryan Donovan speaking with Ahmed Aldali, CTO of Airbnb, about how Airbnb approaches AI engineering in a company with an established machine learning culture. The conversation centers on why customer support is the hardest AI use case, how Airbnb evaluates and improves generative AI systems, and how the company thinks about productivity, quality, and engineering judgment in the age of agentic coding tools.
Key Themes and Takeaways
Why customer support is Airbnb’s hardest AI problem
- Airbnb intentionally chose customer support as the first major generative AI use case because it is one of the most difficult to get right.
- Support at Airbnb is more complex than typical chatbot use cases because:
- It involves a two-sided marketplace: guests, hosts, and Airbnb often all have different interests.
- Many cases are multi-party conflict resolution, not just answering a question.
- The stakes can be high, especially for expensive bookings and sensitive issues.
- The team needs responses to be not just helpful, but correct, policy-compliant, and trustworthy.
Airbnb’s AI foundation
- Airbnb already had a strong machine learning program before generative AI, largely focused on:
- search
- ranking
- recommendations
- That foundation made it easier to expand into generative AI and agents across more of the product and operations stack.
- Airbnb’s AI use cases are highly multimodal, involving text, images, and marketplace context.
Evals are central to AI development
- Aldali repeatedly emphasizes that AI engineering starts with evaluation.
- Airbnb uses production traffic and carefully designed evaluations to measure performance in real scenarios.
- Their philosophy is not to rely on one fixed benchmark. Instead:
- evals are a moving target
- teams should keep “hill climbing” toward harder cases
- if a benchmark is saturated, the eval is no longer useful enough
- A good eval taxonomy should include increasing levels of difficulty and cases where the model should defer to a human.
Quality, latency, and cost must all be balanced
- Airbnb cannot just use the biggest, slowest model.
- Their requirements include:
- strong accuracy
- low latency
- smooth user experience
- global, multilingual deployment
- They aim to create their own Pareto frontier by combining:
- strong internal evals
- specialized models
- infrastructure tuned for their use case
Human judgment matters as much as AI output
- Aldali argues that using AI well is less about automation and more about judgment.
- He warns against “slop docs” and “slop PRs” — artifacts generated by AI that the creator does not understand or defend.
- His core idea: if you use AI to produce something, you are still responsible for it.
- This is true for both senior and junior engineers, but senior engineers often have more judgment from experience shipping and debugging production systems.
Agentic coding boosts productivity, but can hurt coherence
- AI coding tools can increase individual productivity dramatically.
- But they can also reduce team coherence if people produce work they do not fully understand.
- Airbnb is experimenting with ways to encode institutional knowledge into systems, including a kind of context graph or shared memory layer.
- The goal is to make agents more useful while preserving engineering standards and norms.
The challenge of skill sprawl in AI tooling
- Large organizations can end up with thousands of AI-powered “skills” or tools that overlap.
- Aldali sees this as both:
- a sign of healthy experimentation
- a selection problem, because too many similar tools make it hard to know which one should become the standard
- Over time, winners emerge naturally, and those are the systems teams should invest in.
Measuring productivity: outcomes over tokens
- Aldali says token usage is not a useful productivity metric except in one case: zero usage, which may indicate no adoption.
- What matters more is:
- speed of execution
- feature output
- product quality
- lower human coordination overhead
- Airbnb’s internal example: they shipped 80% more features year over year with similar complexity.
Reliability and quality remain non-negotiable
- Aldali is especially concerned that AI speed could increase defects or outages.
- Airbnb tracks reliability closely and uses strong operational controls, including:
- code yellow / code red escalation
- observability
- remediation
- rigorous SEV processes
- He notes that while AI usage has grown geometrically, production defects have only grown linearly, which he sees as a sign of reasonable hygiene so far.
Not all work should be measured the same way
- Velocity depends on the kind of feature being built.
- A small homepage change has a very different “human tax” than a regulated or cross-functional product launch.
- For infrastructure and migrations, AI can make previously “impossible” projects feasible, and those are easier to measure directly.
Notable Ideas and Quotes
- “Customer support is the hardest example that you can pick because you have to be correct.”
- “Evals, evals and evals.”
- “If I see a team tell me we got 95% on this eval, I tell them to redo their evals because that’s saturated.”
- “The difference between using a coding agent well and poorly is whether you apply judgment to that process.”
- “We’re not deploying these systems for the sake of maximizing AI, we’re trying to improve the user experience.”
Main Takeaways for AI/Engineering Teams
- Start with the hardest, most realistic use case if you want to build robust AI systems.
- Treat evaluation as an ongoing discipline, not a one-time benchmark.
- Optimize for quality, reliability, and user trust, not just automation.
- Encourage AI adoption, but create cultural norms that require engineers to understand and defend what AI produces.
- Measure AI value in terms of business outcomes and product velocity, not raw token consumption.
- Expect AI to change productivity, but don’t let it erode engineering rigor.