Summary of No country left behind with sovereign AI

Overview of The Stack Overflow Podcast — "No country left behind with sovereign AI"

Host Ryan Donovan interviews Steve Watt (Distinguished Engineer & VP, Office of the CTO, Red Hat) about "sovereign AI" — what nations and regions are building when they insist on control over AI and data, how the stack and operations differ from standard cloud deployments, and the engineering patterns and trade-offs used to deliver reliable, compliant, and performant AI services inside national boundaries.

Key topics covered

• Definitions and lenses for sovereignty: digital/data sovereignty vs. sovereign cloud vs. sovereign AI.
• Physical constraints impacting sovereign AI: power, cooling (liquid cooling), land, and regional resource limits.
• Differences between traditional cloud-native and AI workloads (pre-training vs. inference).
• Orchestration and runtime choices: Slurm, Kubernetes, and hybrid approaches.
• Techniques to stretch scarce accelerators: model routing, caching, disaggregation, and CPU-based inference.
• Open-source and transparency concerns for model weights, pipelines, and legal risk (copyright/indemnification).
• Future hardware trends: inference-focused accelerators, RISC‑V, and accelerator plug-in architectures for PyTorch.

Main takeaways

• Sovereign AI is more than "put data centers in-country." It combines regulatory guarantees (data residency, operator identity) with infrastructure and operational choices that meet local needs and constraints.
• Pre-training (long, job-oriented workloads) and inference (app-like, SLA-driven workloads) have different operational profiles and often need different tooling (Slurm vs. Kubernetes or both).
• Because AI accelerators are expensive and non‑ephemeral, operators must optimize utilization via caching, request routing, disaggregation of model components, and sometimes CPU fallbacks.
• Transparency matters: “open-weight” alone is insufficient for true open/sovereign AI — the software components, weights, training pipeline, and datasets all factor into trust and legal exposure.
• Geography (land, water for cooling, grid capacity) will shape sovereign AI strategies — some regions may outsource operation or adopt lower-power inference alternatives until local infrastructure is ready.

Technical stack & patterns (practical summary)

Workload split

• Pre-training: large, long-running jobs — typically run on HPC schedulers like Slurm (high node counts) or on large Kubernetes clusters depending on scale.
• Post-training / fine-tuning: often job-based but can use either Slurm or Kubernetes depending on reinforcement learning or continuous inference loops.
• Inference (serving): app-like SLA/SLO requirements favor Kubernetes for reliability, scaling, and operational guarantees.

Orchestration & joint models

• Slurm and Kubernetes can be complementary: Slurm for massive pre-training jobs; Kubernetes for inference and reliably serving model endpoints.
• Slinky: community project that runs Slurm on Kubernetes to combine benefits.

Making most of scarce accelerators

• vLLM (inference servers): typically one model per server instance; scale by spinning up many instances and using load-balancing.
• Semantic routing / inference gateways: route requests to the right model based on content/policy, perform caching of repeated queries to reduce load.
• Disaggregation (example project cited as LLMD): split prefill, decode, and KV-cache across servers to improve throughput and utilization.
• CPU inference options: run lighter inference on existing x86 fleets where latency/throughput requirements allow; modern CPUs include matrix/matrix‑friendly instructions to help.
• Aggregate many CPU-backed instances via Kubernetes to meet token throughput targets when GPUs aren’t available.

Constraints, risks, and trade-offs

• Physical limits: power grid capacity, water supply (for liquid cooling), and available land materially affect where and how you can deploy high-density AI infrastructure.
• Sovereign paradox: if a region lacks local capacity, it may run “sovereign” services in other regions (Nordics, etc.), undermining strict national sovereignty.
• Legal & transparency risks: open-weight models without disclosed pipelines/datasets present copyright and regulatory exposure. Some vendors (example: IBM Granite) offered indemnification to address legal risk.
• Fragmented maturity: users range from experimentation-level teams to enterprises moving workloads back on-prem — each has different needs and constraints.

Recommendations / action items for implementers

• Design sovereignty as both policy and service: create a user-facing function to onboard and subsidize local researchers/startups, and a separate operations function (or partner) to run infrastructure.
• Choose orchestration based on workload profiles: use Slurm for extreme-scale pre-training, Kubernetes for inference (SLA-driven), and consider hybrid or integration projects (e.g., Slinky).
• Optimize scarce accelerators: implement semantic routing and caching, disaggregate LLM internals where useful, and consider CPU inference for lower-latency-tolerant workloads.
• Favor reproducibility and transparency: aim to document pipelines, datasets, model weights and software components if you require true "open" or legally defensible AI.
• Plan for hardware diversity: design software so accelerator support is plugin-based (recent changes in PyTorch make this easier), enabling adoption of inference-focused accelerators or RISC‑V implementations.

Notable quotes and insights

• "Sovereign AI is way different" — physical infrastructure (power, cooling, land) makes sovereign AI materially harder than simple data residency.
• "Inference is way more like an app" — inference needs operational guarantees (SLA/SLO) in a way model training jobs do not.
• "Sunlight is the best disinfectant" — real openness should include pipeline and dataset transparency, not just published model weights.
• "AI is speed-running the path to microservices" — agentic systems are similar to microservices; lessons from earlier service architectures (SOAP→REST) should inform new designs.

Projects, companies, and technologies mentioned

• Orchestration/schedulers: Kubernetes, Slurm, Slinky (Slurm-on-Kubernetes).
• Inference & model tooling: vLLM, LLMD (disaggregation), VLM (CPU inference concept), semantic router/inference gateway.
• Hardware/accelerators: GPUs (NVIDIA/AMD), Groq, Cerebras, RISC‑V inference chips.
• Open-weight / model providers: mention of region-specific alternatives (example: Reflection AI); IBM Granite as an example of indemnified open model.
• Geographic examples: Saudi Arabia and UAE (early sovereign AI adopters), Nordics/Finland (data center heat reuse).

Bottom line

Sovereign AI is a socio-technical program: it couples regulatory requirements and regional policy with deep engineering trade-offs. Delivering reliable, sovereign AI requires rethinking orchestration (Slurm vs Kubernetes), optimizing scarce accelerators via routing/caching/disaggregation, embracing pipeline transparency for trust and legal risk mitigation, and designing for the physical realities of each region (power, cooling, land).