What's left for infrastructure-as-code after AI moves in?

Summary of What's left for infrastructure-as-code after AI moves in?

by The Stack Overflow Podcast

30m•July 8, 2026
Listen to Episode

Overview of What's left for infrastructure-as-code after AI moves in?

In this Stack Overflow Podcast episode, host Ryan Donovan speaks with Rosemary Wang, developer advocate at IBM, about how AI is changing DevOps and infrastructure as code (IaC). The big idea: AI is not eliminating DevOps so much as reshaping it. As coding agents make it easier for more people to generate and deploy software, platform engineering, security, and policy enforcement become even more important. The conversation focuses on guardrails, context engineering, token costs, and how teams can keep AI-assisted infrastructure safe, reliable, and affordable.

Key Themes and Takeaways

AI is making more people “closer to DevOps”

  • AI lowers the barrier to creating and deploying software, even for people who aren’t infrastructure experts.
  • That empowerment is useful, but it also increases the risk of deploying things without understanding the operational impact.
  • Rosemary argues that AI is making DevOps more accessible, not obsolete.

Platform engineering, security, and AI are deeply overlapping

  • A major theme is that AI concerns can’t be treated in isolation.
  • Examples:
    • RAG / knowledge retrieval touches AI, platform engineering, and security.
    • Vector databases and knowledge graphs are platform concerns.
    • Data sanitization, encryption, and redaction are security concerns.
  • The takeaway: AI capabilities should be moved into platform thinking, not treated as a standalone magic layer.

Context is the new bottleneck

  • Rosemary emphasizes that AI systems are only as good as the context they receive.
  • But context is expensive:
    • Too much context increases token usage and cost.
    • Poorly chosen context can leak sensitive information or inject outdated practices.
  • She describes the “harness” as the combination of rules, tools, skills, and policies that make an agent useful and safe.

Guardrails matter more than raw capability

  • The episode repeatedly returns to the need for strong guardrails:
    • Approved packages only
    • Specific provider/version restrictions
    • Compliance requirements
    • Human-in-the-loop review where needed
  • A recurring concern is that if instructions are too vague or context is compressed too aggressively, an agent may misunderstand or drop important constraints.

Deterministic tools still matter

  • Rosemary makes a strong case that not everything should be handled by an LLM.
  • Examples of deterministic tools that should stay in the workflow:
    • Linters
    • Testing
    • Policy-as-code
    • Cost optimization checks
  • These tools handle “known knowns” better than a model and reduce unnecessary token spending.

Token costs are now a real operational concern

  • The guests discuss how AI token usage has become a budget issue.
  • Organizations are moving from subsidized model usage to real procurement pressure.
  • This is pushing teams toward:
    • Model routing
    • Small language models (SLMs) for simple tasks
    • Larger frontier models only when needed
    • Speculative decoding and other cost-optimization techniques

Infrastructure as Code in the Age of AI

AI can generate IaC, but debugging and safety are harder

  • Rosemary says agents can often generate infrastructure code, but that doesn’t mean they can reliably debug it.
  • IaC is especially risky because mistakes can have a large blast radius.
  • A bad deployment isn’t just a broken app; it can take down an environment or create security/compliance problems.

Terraform and DSLs are especially tricky

  • Terraform’s domain-specific syntax makes it easy for models to hallucinate outdated or invalid patterns.
  • Older examples in training data can lead agents to generate deprecated code.
  • For DSL-heavy infrastructure workflows, the answer is not necessarily a custom model, but better:
    • Harnesses
    • RAG
    • Tooling
    • Explicit specs and policies

Best practice: keep infrastructure tasks modular and explicit

  • Don’t ask an agent to solve an entire infrastructure problem at once.
  • Break tasks into smaller pieces with clear dependencies and rules.
  • Be very specific about:
    • Where the model should get information
    • Which tools it should use
    • What it must never do

Security and Access Control Are Under Pressure

Vibe coding creates new failure modes

  • The episode includes an example of a vibe-coded application reaching production through an auto-merge path.
  • This leads to a strong warning: if an agent can bypass approvals, the real issue is access control and policy design, not the model alone.

Least privilege needs rethinking in agentic systems

  • Traditional security assumptions break down when agents can act autonomously.
  • Rosemary suggests that the industry still doesn’t have perfect answers here, but the need for:
    • logging
    • auditing
    • strong approvals
    • escalation paths
      is now obvious.

Practical Guidance for Teams

For platform and DevOps teams

  • Treat AI as part of the platform, not a separate experiment.
  • Build a strong harness around agents:
    • approved tools
    • policy libraries
    • best-practice templates
    • compliance references
  • Use deterministic checks wherever possible before involving an LLM.

For organizations adopting coding agents

  • Be specific about rules and constraints.
  • Feed in curated best practices from cloud providers and internal policy teams.
  • Use smaller models for simple, repetitive tasks.
  • Reserve larger models for genuinely complex work.
  • Ensure there is always a human review path for risky changes.

For teams working with IaC

  • Modularize infrastructure code generation.
  • Avoid asking an agent to produce massive, environment-wide changes in one shot.
  • Expect agents to help with creation more than debugging.
  • Watch for outdated Terraform syntax and legacy patterns.

Notable Insight

“We’re not looking for autonomy, we’re looking for empowerment.”

This is the episode’s clearest takeaway: AI should help more people do useful work safely, but it should not replace the discipline, review, and operational knowledge that keep systems reliable.

Closing Notes

The episode ends on a practical note: DevOps and platform engineers are not disappearing. Their role is shifting toward enabling, curating, and distributing expertise so that AI-assisted development doesn’t become a security, compliance, or reliability nightmare. The future of infrastructure-as-code is less about replacing humans and more about building better systems around them.