Summary of Building Durable AI Agents

Overview of Building Durable AI Agents

This episode of the Practical AI Podcast explores how to build durable, reliable AI agents that can operate beyond a developer’s laptop and into real production environments. Host Daniel Whitenack talks with Hamza Tahir, co-founder of ZenML and creator of Kitaru, about how lessons from MLOps are resurfacing in the agent era—especially around workflows, orchestration, retries, state, observability, and safe execution of non-deterministic code.

The core theme: agents are not just prompts and models. They are systems with state, dependencies, tool calls, failure modes, and long-running execution paths that need production-grade infrastructure.

Key Takeaways

• Agents are workflows under the hood.

  • Even when users experience an agent as “free-form,” it still executes as a graph of model calls, tool calls, branching, and state transitions.
  • The big shift is from static DAGs to dynamic, runtime-defined graphs.

• Durability is now the main challenge.

  • Agents are often fragile because they run long, stateful, and non-deterministic processes.
  • Common production needs include:
    • retries
    • checkpointing
    • replay
    • idempotency
    • observability
    • state persistence

• The harness matters as much as the model.

  • Hamza defines the harness as the software layer that turns model tokens into actions: tool calling, memory, compaction, context handling, and control flow.
  • The agent is effectively the combination of model + harness.

• Model and harness are becoming coupled.

  • Model providers are increasingly optimizing for their own harnesses, making the stack more integrated and harder to swap.
  • This creates tension between:
    • proprietary, model-tied agent systems
    • open harnesses and portable runtimes

• Enterprise agents need infrastructure, not just prompts.

  • As agents move from local use to cloud and enterprise deployment, they need the same rigor as production services.
  • That includes queues, workers, sandboxes, versioning, and runtime isolation.

What Makes AI Agents Fragile?

Hamza outlines several failure modes and architecture pain points that make agents hard to productionize:

Infrastructure and runtime issues

• long-running tasks that can’t safely stay in-process
• worker failures and pod restarts
• network interruptions
• sandbox execution for arbitrary code
• file-system state loss during code generation or editing
• hard-to-debug failures in tool execution

Agent-specific issues

• changing prompts or tool definitions can alter behavior unpredictably
• tool calls may time out or fail mid-loop
• model changes can break previously working traces
• output quality is difficult to test and compare rigorously
• updates are risky because the system is highly stateful

Scale-related issues

• once agents run in fleets or swarms, the volume and complexity grow quickly
• enterprise use cases can involve many simultaneous executions, versioned workflows, and long-lived processes
• teams need to understand what happened in each trace and how to improve it

What Kitaru Is Trying to Solve

Kitaru is Hamza’s new project built on top of ZenML, designed as an open runtime for agents.

Core goals of Kitaru

• support popular agent harnesses such as:
  • Anthropic Agents SDK
  • OpenAI Agents SDK
• move agent execution from local/dev environments into production
• add durability through:
  • checkpointing
  • external state storage
  • replay
  • trace inspection
• make it easier to test alternate choices, such as:
  • different models
  • different tool sets
  • different prompt strategies
  • different harness behaviors

Why replay matters

• Replay allows teams to inspect what happened and experiment with alternatives.
• You can learn, for example:
  • whether a cheaper model would have worked
  • whether fewer tools would have reduced confusion
  • whether a different execution path would have been faster or more reliable
• Hamza argues that replay is essential because it makes agent optimization measurable instead of purely anecdotal.

Recommended Architecture Direction

For teams building agents for production, the conversation suggests a few practical principles:

• Separate the agent harness from infrastructure concerns

  • treat execution, persistence, and recovery as first-class system concerns

• Use queues/workers for long-running tasks

  • don’t run serious agent loops directly inside request handlers

• Checkpoint early and often

  • store state so you can recover from tool failures or interrupted execution

• Build for observability

  • keep traces, inspect bottlenecks, and analyze failures

• Expect updates to be risky

  • versioning and replay are necessary because even small prompt or tool changes can affect behavior

• Own the platform if agents are core to the business

  • for companies deploying agents at scale, internal platform investment can become a competitive advantage

Broader Industry Outlook

What Hamza is optimistic about

• Open-source and open-model momentum

  • improving open models make it more viable for enterprises to own more of the stack
  • examples mentioned include models like GLM, Kimi, and Minimax

• Open harnesses

  • frameworks like Pi, LangGraph, and PydanticAI may become increasingly important
  • specialized harnesses for verticals like law or science could also grow

• Enterprise differentiation through infrastructure

  • as model performance commoditizes, the real moat may shift to:
    • internal agent platforms
    • durable runtimes
    • workflow intelligence
    • operational know-how

What he’s less optimistic about

• too many competing frameworks
  • the ecosystem feels early and fragmented
• proprietary lock-in
  • model providers increasingly shape how agents are deployed, which can reduce interoperability
• unclear canonical best practices
  • the industry is still figuring out the “right” way to separate harness, runtime, and deployment layers

Practical Advice for Builders

If you’re building agents today, the episode suggests:

• start by understanding your workload type:

  • chatbot
  • voice agent
  • research agent
  • autonomous background task
  • business-process agent

• use managed tooling if the use case is simple

• invest in internal platforming once the agent becomes critical or high-scale

• keep traces and checkpoints from day one

• design for replay, inspection, and model swapping

• expect to iterate on the harness, not just the prompt

Notable Insight

“Everything is a workflow; an agent is just an unrolled graph.”

That idea captures the episode’s main point: agent systems may look magical at the surface, but durable production systems still depend on classic engineering fundamentals—state, orchestration, reliability, and observability.