You need reliable AI context for your site reliability

Summary of You need reliable AI context for your site reliability

by The Stack Overflow Podcast

26m•July 28, 2026
Listen to Episode

Overview of You need reliable AI context for your site reliability

In this Stack Overflow Podcast episode, Ryan Donovan speaks with Asaf Savic, AI Engineering Group Manager at Commodore, about how AI can support Site Reliability Engineering (SRE) without overwhelming engineers with too much data. The conversation centers on context engineering: how to gather, filter, and evaluate the right information so AI agents can help investigate incidents, propose fixes, and gradually move toward safer automation in production environments.

Key Themes and Takeaways

What SRE does and why AI helps

  • SREs are responsible for keeping production systems stable, healthy, and reliable.
  • Their work requires understanding:
    • infrastructure
    • microservices relationships
    • logs, metrics, and traces
    • code changes and deployment history
    • business and product context
  • As systems grow more complex, SREs carry more responsibility and cognitive load, making AI assistance especially valuable.

Context engineering is the real challenge

  • The team initially tried a naïve approach: give the AI lots of context and see what happens.
  • That quickly became inefficient and unreliable.
  • The main challenge is not just collecting data, but deciding:
    • what information is critical
    • what should be summarized
    • what should be injected upfront vs. fetched later
  • Two core pillars emerged:
    1. Data collection and injection
    2. Evaluation/testing to ensure changes actually improve outcomes

Why “more context” is not always better

  • Production incidents can involve:
    • dozens of services
    • noisy logs
    • changing metrics
    • hidden dependencies across teams
  • Too much context increases:
    • token usage and cost
    • latency
    • hallucination risk
    • chance of degraded accuracy
  • Savic argues that for production incident handling, AI must be extremely accurate, not just “pretty good.”

How Commodore Structures Its AI SRE System

Main agent + sub-agents

  • Rather than forcing one general-purpose agent to do everything, Commodore uses:
    • a main agent for orchestration and incident understanding
    • sub-agents for focused tasks like log summarization
  • Sub-agents receive only the context they need and return concise results to the main agent.
  • This improves:
    • speed
    • cost efficiency
    • reliability
    • consistency
    • reduced hallucinations

Model selection and optimization

  • They use off-the-shelf models, especially Claude.
  • Different tasks may use different models depending on complexity and cost.
  • Even when a model can handle very large context windows, they still optimize aggressively because:
    • more context can reduce accuracy
    • performance matters in live incidents
    • the goal is near-perfect reliability in production

Evaluation: The Foundation of Trust

Why evaluation is essential

  • Savic emphasizes that good judges/evaluations are the baseline for everything.
  • Every change is tested against real or simulated incidents to confirm it improves results.
  • They compare:
    • accuracy
    • token usage
    • speed
    • consistency

How they judge performance

  • They built their own evaluation system and improved it over time through manual review.
  • A/B comparisons are more reliable than asking for a raw score.
    • It’s better to ask: “Which is better, A or B?”
    • than: “Rate B from 1–100.”
  • Static evaluation inputs make it easier to improve judges, unlike real incidents, which are dynamic and changing.

Regression testing with scenarios

  • Because they can’t use customer data directly, they anonymize real issues into scenarios.
  • These scenarios simulate production environments with:
    • noisy services
    • irrelevant alerts
    • multiple pods, databases, Redis, AWS services, etc.
  • They break the setup intentionally, run agents against it, and test daily for regressions.

Remediation and the Human-in-the-Loop Approach

Autonomous fixes are possible, but trust is still the barrier

  • Commodore’s agent, Claudia, can:
    • investigate incidents
    • propose a remediation plan
    • recommend the exact command or action to take
  • But in most production environments, full autonomy is still rare.
  • The current preferred model is human-in-the-loop:
    • AI proposes
    • a person approves or rejects
    • if approved, the agent can execute the fix

What kinds of fixes are safest

  • Users are more comfortable with limited-autonomy actions like:
    • rollbacks
    • low-risk operational changes
  • More dangerous actions, such as code changes or new deployments, still require stronger trust and review.

Preventive and Postmortem Use Cases

The next phase is prevention

  • The conversation suggests the next wave of AI SRE will move from:
    1. detection
    2. investigation
    3. remediation
    4. prevention
  • AI can analyze recurring issues to find patterns in:
    • databases
    • networking
    • connections
    • GitOps/Helm/Argo deployments

Postmortem generation and institutional memory

  • The system can help users revisit resolved incidents and generate postmortem guidance.
  • It can learn from:
    • uploaded knowledge bases
    • prior postmortems
    • a memory layer of past investigations
  • This helps the AI improve future investigations and close the loop after incidents are resolved.

How the Product Is Evolving

From UI usage to agentic integration

  • Initially, users interacted with Claudia primarily through the Commodore UI.
  • More recently, usage has shifted toward:
    • API
    • MCP
    • Slack
    • custom orchestrator agents
  • In some cases, users may not even realize Claudia was triggered behind the scenes.
  • This reflects a broader shift toward AI as a backend capability inside other workflows.

Build vs. buy in the AI era

  • The episode touches on the renewed interest in open source and custom-built systems.
  • But Savic argues that production incident response is not the place for a hobby project:
    • reliability matters too much
    • enterprise support matters
    • real incidents demand professional-grade tooling

The Future of SRE

What SREs may focus on next

Savic believes SREs will still handle incidents, but their role will expand toward:

  • cost optimization
  • planning ahead
  • strategic reliability work
  • less time firefighting, more time improving systems

AI as the incident lead

  • He envisions AI leading investigations while humans become:
    • managers of agent workflows
    • approvers and reviewers
    • strategic decision-makers
  • In this model, AI reduces the constant pressure on SREs while helping them work more effectively.

Notable Insights

  • “Just throwing context at the problem” doesn’t work in production incident response.
  • Accuracy must remain extremely high in SRE tooling; “good enough” is not enough.
  • A/B testing and strong judges are essential for trustworthy AI systems.
  • Human-in-the-loop remains the default for high-risk remediation.
  • The most promising future use case is preventive AI SRE, not just reactive incident handling.

Company / Product Mentioned

Commodore and Claudia

  • Commodore builds AI tools for SRE and incident response.
  • Its AI agent, Claudia, is used for:
    • investigation
    • remediation planning
    • prevention
    • postmortem support
  • More information was shared at commodore.io.