Summary of Building resilient systems with Sam Newman

Overview of Building resilient systems with Sam Newman

This episode is a deep conversation between Gergely Orosz and Sam Newman, best known for Building Microservices and now Building Resilient Distributed Systems. The discussion covers how Sam’s thinking evolved from microservices and continuous delivery into resilience engineering, and how those ideas apply to modern software development, AI-assisted coding, observability, and system design. A recurring theme is that resilient software is not just about code—it’s about people, trade-offs, production reality, and knowing what “good” looks like.

Sam Newman’s background and how he got into software

Early computing and education

• Sam got interested in computers early through home machines like the Acorn Electron and ZX Spectrum.
• He studied software engineering at De Montfort University in Leicester.
• A “sandwich course” year in industry gave him real-world experience and clarified his career path.

Early professional lessons

• His first major work involved maintaining large Fortran 77 codebases for scientific/engineering systems.
• One formative moment was refactoring thousands of loop limits using regular expressions, which gave him an early sense of the power of tooling and automation.
• He emphasizes that software is always a field of reinvention and that engineers regularly get humbled by better ways to solve old problems.

ThoughtWorks and the rise of modern delivery practices

What he did there

• Sam spent many years at ThoughtWorks, where he worked on:
  • continuous integration and build pipelines
  • test automation
  • infrastructure automation
  • delivery optimization
  • early continuous delivery practices

What he learned

• ThoughtWorks was a place where ideas were constantly challenged and improved.
• He describes the environment as one where “if you need it, just do it” was a real mindset.
• The company helped shape his thinking on:
  • pairing
  • TDD
  • fast feedback loops
  • architecture as something that can be changed, not fixed

Microservices: why Sam calls them an “architecture of last resort”

Origin of the term

• The term microservices was coined in a ThoughtWorks/London-area context, with James Lewis playing a central role.
• The concept emerged from a wider movement around:
  • service-oriented architecture (SOA)
  • REST
  • continuous delivery
  • independent deployability

What microservices are really for

• Sam’s strongest view is that microservices are valuable when you need:
  • independent deployability
  • strong team autonomy
  • organizational decoupling
• He sees them as a tool to solve organizational and delivery problems, not as a goal in themselves.

Why “last resort”

• Microservices introduce:
  • distributed state
  • operational complexity
  • increased failure modes
• In practice, many teams adopt them because the monolith is blocking delivery, not because microservices are inherently superior.
• He argues that many “microservice” systems are really just fine-grained SOA with a new label.

Domain boundaries vs technical boundaries

• Sam prefers boundaries based on the business domain rather than technical layers.
• But he notes that in some tech companies, services emerge more organically and less formally around the product structure.

Production is truth: specs, code, and source of truth

Code vs spec

• The episode discusses the growing debate over whether the source of truth is:
  • the code
  • the spec
  • or production itself

Sam’s view

• His blunt position: production is truth.
• Specs are useful, code is useful, but production reveals what the system actually does.
• He is open to spec-driven development, but only if teams can clearly define:
  • what “good” looks like
  • how to verify it
  • how the spec stays aligned with reality

The practical concern

• If teams stop looking at code without replacing that work with deeper system thinking, they risk losing the shared mental model.
• He believes spec-first approaches can work, but only if they improve collaboration and verification rather than becoming a form of “AI-generated hand-waving.”

Idempotency: retries, fingerprints, and why keys are better

What idempotency means

• An idempotent operation can be repeated safely without changing the outcome beyond the first execution.
• Good examples:
  • pressing a lift button
  • many delete operations
• Bad examples:
  • toggles
  • charging a card twice

Why it matters

• In distributed systems, requests may be retried because:
  • the response was lost
  • the target was unavailable
  • the network was slow or partitioned
• If retries are not safe, you can create duplicate side effects.

Two approaches

• Idempotency keys:
  • Best approach when building a system properly
  • Requires client and server support
  • Safer and more explicit
• Fingerprints:
  • Useful for retrofitting idempotency
  • Can cause false positives when legitimate repeated requests look identical
  • Often less reliable than true keys

The three rules of distributed systems

Sam boils distributed systems down to three core realities:

1. It takes time

   • Nothing moves instantly across a network.
   • Latency is unavoidable.

2. Sometimes the thing you want to talk to isn’t there

   • Services fail, disappear, restart, or become unreachable.

3. Resource pools are not infinite

   • CPU, memory, queue depth, and I/O all saturate.
   • Many outages are ultimately saturation problems.

These three rules are the foundation for understanding retries, timeouts, load balancing, and failure behavior.

Thundering herds, retries, and overload

What a thundering herd is

• A thundering herd happens when too many requests hit too few resources.
• Common causes include:
  • cache collapse
  • retry storms
  • DDoS attacks
  • sudden viral success

Why it matters

• Retries can make outages worse if they are too aggressive.
• A retry storm can itself become the outage.

Mitigations depend on the cause

• For malicious traffic: rate limiting, edge protection, and services like Cloudflare
• For cache collapse: warming or staging caches
• For success spikes: autoscaling, throttling, admission control
• For retry storms: better backoff, jitter, and fewer hard-coded retry loops

Resilience engineering: the four dimensions

Sam frames resilience using four dimensions, inspired by resilience engineering research:

1. Robustness

• Absorbing known disturbances without user-visible failure.
• Example: Kubernetes restarting a failed pod.

2. Rebound

• How quickly the system recovers after a failure.
• This includes detection, recovery, and restoration.

3. Graceful extensibility

• How well the system and organization handle surprise.
• This is about adapting when something happens that wasn’t anticipated.

4. Sustained adaptability

• The ability to keep learning and evolving over time.
• Requires feedback, incident analysis, and a culture that shares lessons.

Key point

• Resilience is not binary.
• It is a degree and a trade-off, not a checkbox.

Observability, SLOs, and knowing what “good” looks like

Why observability matters

• Distributed systems are hard to understand because signals are spread across many machines.
• Observability is the ability to infer internal state from external signals.

What good observability looks like

• Logs
• Metrics
• Traces
• Structured events
• Service level indicators/objectives:
  • SLIs: measurements of system behavior
  • SLOs: targets for acceptable performance

Why it’s essential

• Without observability, teams can’t:
  • set sensible timeouts
  • know if retries are safe
  • understand failure patterns
  • define “good enough” for the business

Trade-offs, business context, and fail-open vs fail-close

The core idea

• Engineering choices must reflect business context.
• The same technical failure may require different responses depending on the product.

Examples

• E-commerce may fail open:
  • sell now, refund later if inventory turns out wrong
• Ticketing / high-stakes booking may fail closed:
  • do not sell if certainty is insufficient

Why this matters

• Teams need to discuss:
  • customer impact
  • revenue impact
  • operational cost
  • latency implications
  • legal or reputational risk

Big takeaway

• Engineering resilience is as much about decision-making as it is about technology.

AI, cognitive debt, and cognitive surrender

Sam’s view of AI in software

• He is pragmatic rather than hype-driven.
• AI is useful at the micro level:
  • code generation
  • analysis
  • summarization
  • incident support
  • documentation and research assistance

What worries him

• AI can increase:
  • context switching
  • complexity
  • loss of shared understanding
• He warns about:
  • cognitive debt: erosion of shared mental models and critical thinking
  • cognitive surrender: handing judgment over to the AI without real review

His practical advice

• Use AI where it helps, but:
  • stay connected to the system
  • keep humans in the loop where judgment matters
  • avoid making AI a black box for core architecture decisions
  • prefer deterministic code where possible

Model and vendor hedging

• He recommends being:
  • multi-vendor
  • multi-model
  • ready to swap out AI components
• He does not believe every workflow should become fully agentic.

How to use AI responsibly in system design

Good use cases

• Research assistance
• Log and incident analysis
• Drafting support
• Controlled workflows
• Code generation inside well-defined modules

Bad use cases

• Over-trusting AI on critical decisions
• Letting AI replace expertise
• Using AI to generate work nobody will review
• Building overly complex workflows around uncertain capabilities

His architectural recommendation

• Keep the system architecture in your head and in the code/specs
• Let AI operate inside clearly defined boundaries
• Spend human effort on:
  • module boundaries
  • system interactions
  • business trade-offs
  • customer impact

Key takeaways

• Microservices are not the goal; independent deployability and organizational autonomy are.
• Production is truth; everything else is a model.
• Distributed systems are hard because time, absence, and resource limits are unavoidable.
• Idempotency keys are better than fingerprints when you can use them.
• Resilience requires observability, not guesswork.
• Trade-offs depend on business context; the same failure can justify fail-open or fail-close behavior.
• AI is useful, but not magical; use it selectively and defensively.
• Resilient systems are socio-technical systems; culture and psychological safety matter.

Resources and further reading mentioned

• Sam Newman:
  • Building Microservices
  • Building Resilient Distributed Systems
  • samnewman.io
• Thought leaders/books referenced:
  • Martin Fowler
  • Kent Beck
  • David Woods on resilience engineering
  • Neil Ford and Mark Richards on software architecture
  • Vlad Kononov on coupling and DDD
  • Parnas papers on modularity and information hiding
• Concepts/tools referenced:
  • OpenTelemetry
  • SLOs / SLIs
  • chaos engineering
  • incident postmortems
  • spec-driven development tools like Kiro / SpecKit