Summary of Code isn’t the only thing causing your production failures

Overview of Code isn’t the only thing causing your production failures

This episode of the Stack Overflow Podcast features Anish Agarwal, CEO and co-founder of Traversal, discussing why modern production failures are increasingly about system complexity, observability gaps, and AI-generated code behavior—not just bad code. The conversation centers on how AI changes both the pace of code creation and the difficulty of understanding what happens once that code reaches production.

Key Themes and Takeaways

• AI makes code easier to generate, but harder to reason about

  • Code generated by copilots and agents can look correct while subtly reducing the developer’s understanding of the system.
  • The main challenge is no longer just writing code, but predicting how it behaves once it interacts with real-world production systems.

• Production failures often emerge at system boundaries

  • Even “correct” code can fail when it interacts with other services, dependencies, or external conditions.
  • Agarwal compares this to self-driving cars: the system may be well-specified, but reality introduces unpredictable interactions.

• Troubleshooting is getting harder as systems scale

  • Cloud-native architecture, microservices, and AI-driven complexity have exploded the amount of telemetry.
  • Meanwhile, the supply of highly skilled SREs has not kept pace, making incident response more difficult.

Observability in the Age of AI

Why traditional observability is not enough

• Current observability tooling is largely built for dashboards and human-driven investigation.
• Logs, metrics, and traces are often:
  • pre-selected by humans,
  • stored in silos,
  • optimized for known queries rather than agentic search.

What AI changes

• AI systems can search through huge volumes of telemetry and find patterns humans may miss.
• LLMs are especially good at what Agarwal calls “soft joins”—connecting semantically related signals across logs, metrics, traces, and other data sources.

The key infrastructure shift

• The problem is not just data volume; it’s also how the data is indexed, cached, and queried.
• Production data systems need to be restructured for agentic, high-volume, exploratory querying, not just dashboard refreshes.

Traversal’s Approach: A Production “World Model”

Traversal’s core idea is to build a causal map of production systems—a “production world model” that shows how changes in one part of the system affect others.

What it does

• Mines production telemetry and related data to infer:
  • causal dependencies,
  • likely root causes,
  • mitigation paths,
  • how incidents propagate across services.
• Uses data from:
  • observability platforms,
  • APIs,
  • raw streams,
  • and tools like Jira, Confluence, Slack, GitHub, and streaming infrastructure such as Kafka / OTel forwarders.

Why it matters

• The model reduces redundancy in massive telemetry streams.
• It improves:
  • signal-to-noise ratio,
  • confidence in root-cause analysis,
  • and the economics of using AI on production data.

Do Companies Need More Instrumentation?

A notable point from the discussion: usually, no.

• For large enterprises, the existing observability data is often enough.
• The issue is less about collecting more data and more about:
  • interpreting it better,
  • connecting signals across systems,
  • and using AI to search the data more effectively.
• In many cases, the data already contains enough signal to identify root causes and multiple corroborating paths to the same answer.

AI Codegen Failure Modes

Agarwal highlighted a few ways agentic code generation can go wrong:

• Over-permissioning

  • Agents are often given too much access too quickly in the rush to adopt AI.
  • This can cause unsafe or unexpected behavior.

• Seemingly correct code that fails in production

  • Code can read well to a human reviewer and still break under real conditions.
  • The danger is that AI-generated code often hides its failure modes until runtime.

• Misaligned objectives

  • Agents can become very persistent and optimize for the wrong goal if the prompt or objective is imperfect.
  • This is useful for execution, but risky if the target is poorly defined.

• Novel failure patterns

  • The failures are not just “old dumb code” repeated at scale.
  • They often arise from the difference between what the user meant and what they explicitly asked for.

The Future of SRE: Self-Driving Production

Traversal frames its vision as self-driving production, borrowing the autonomy levels model from autonomous vehicles.

Levels of autonomy

• L0: Manual troubleshooting
• L1: Agents execute existing runbooks
• L2: Agents summarize incidents, find similar historical cases, and generate queries
• L3: Agents troubleshoot bounded classes of issues
• L4: Agents identify mitigation steps across systems
• L5: Fully self-healing production, where the system diagnoses and remediates itself

What’s realistic today

• Traversal claims to already operate around L4 in some contexts:
  • finding root causes,
  • suggesting mitigations,
  • creating tickets,
  • and sometimes executing fixes.
• The remaining challenge is less about technology and more about change management and trust.

Important Practical Insight

• The path to autonomous remediation should be gradual:
  • start with suggestions,
  • move to human approval,
  • then limited commands,
  • and eventually broader automated actions.
• Before code even reaches production, AI can help forecast how it will behave and harden it in advance.

Final Takeaways

• Production failures are becoming a systems problem, not just a code problem.
• Traditional observability is necessary but insufficient for AI-driven troubleshooting.
• Existing telemetry is usually enough if AI can search and connect it intelligently.
• The future of SRE is increasingly agentic, but full autonomy will depend on better evaluation, trust, and organizational readiness.
• The ultimate goal is a self-healing production loop: write code, predict failure, deploy, monitor, and automatically improve.

About the Guest

Anish Agarwal is the CEO and co-founder of Traversal. He shared his background in:

• computer science at Caltech,
• a PhD at MIT focused on AI/ML,
• research in causal machine learning and reinforcement learning,
• and faculty work at Columbia before returning to company building as AI rapidly accelerated.