Summary of The fastest agent in the race has the best evals

Overview of The Stack Overflow Podcast — "The fastest agent in the race has the best evals"

This episode features Benjamin Klieger, AI lead (agents) at Grok, discussing how to build practical, high‑performance agent systems. Rather than focusing on base model architectures, the conversation centers on the surrounding infrastructure: making inference and tools fast, designing agent frameworks, and running meaningful, reproducible evals (including real‑time evals). The guest describes Grok’s agent platform Compound, their inference approach (custom chip + software techniques), and best practices for measuring agent quality and cost‑efficiency.

Guest background

• Benjamin Klieger — AI lead working on agents at Grok.
• Came from a product-first background, moved into engineering and hands‑on model/agent work.
• Focuses on building fast, efficient agent systems and reproducible evaluation infrastructure.

Key topics discussed

• What an agent platform should provide (model abstraction, tool orchestration, routing to best models).
• Importance of speed and cost efficiency (latency matters for UX and quality).
• Architectural choices for agent frameworks: high‑level abstractions vs. low‑level efficient primitives.
• Evals: shortcomings of static benchmarks and the need for reproducible, dynamic testing (OpenBench / real‑time evals).
• Techniques to improve intelligence per second and per dollar (speculative decoding, prefix caching, prompt compaction, delegation/sub‑agents).
• Engineering for cloud infrastructure: scheduling, batching, and compute utilization.
• Practical guidance for builders and where agents are headed.

What is Compound (agent platform)

• An agent that you call like a model — swap one line of code for an agent model id and get agent capabilities.
• Compound can:
  • Search the web iteratively and in real time,
  • Spin up browser windows / query external knowledge engines (e.g., Wolfram Alpha),
  • Execute code,
  • Route sub‑tasks to the best model(s) for the job.
• Design goal: model‑agnostic from the user API while routing to the right models internally for speed/cost/quality.

Building efficient agents — core components

Fast inference

• Latency of underlying models dominates perceived speed: if the model takes 20–30s, you can’t make the agent feel real‑time.
• Grok combines custom hardware (an "LPU" chip per the discussion) with software techniques to speed inference.

Fast tools

• Tool endpoints (search, browsers, APIs) can themselves introduce multi‑second latencies. Optimizing or partnering with tool providers is crucial.

Parallelization & delegation

• Agents can spin up sub‑agents or sub‑tasks to parallelize work (e.g., one sub‑agent per provider to compare results).
• Parallelism + fast inference/tools reduces overall latency for multi‑step reasoning.

Smart model routing

• Different models excel at different tasks — route choose models for search, math, reasoning, etc., to optimize quality and cost.

Minimal abstraction vs efficiency

• High‑level agent frameworks (e.g., LangChain style) can simplify development but introduce extra latency and token costs.
• There’s growing interest in lower‑level, efficient frameworks (and SDKs from providers) that avoid unnecessary overhead.

Evals: problems, solutions, and best practices

Problems with common/static evals

• Benchmarks like SimpleQA (4,000 trivia questions) get saturated (95–99%), are outdated, and fail to capture dynamic/real‑time search needs.
• Eval results across labs are often not comparable due to differing harnesses, hyperparameters, or implementation details.
• Datasets can contain errors; many “hot” evals need verification and cleaning after publication.

Solutions and Grok’s approach

• OpenBench (open‑source project mentioned): a standardized repository and harness to reproducibly run evals.
• Real‑time evals: dynamically create test sets from recent news/RSS feeds so queries represent novelty and current events. Run competitors at the same time to compare fairly.
• Evaluate entire agent systems (model + tools) end‑to‑end, not just base model benchmarks.
• Use evals both for benchmarking and as verifiers in RL training (i.e., to train toward the eval metric).

What to measure

• Traditional quality metrics (accuracy/utility).
• Intelligence per second and intelligence per dollar — measure quality normalized by latency and cost to reflect production tradeoffs.

Techniques to maximize intelligence per dollar/second

• Speculative decoding: use a fast draft model to propose tokens, validate them with the full model to speed generation while preserving quality.
• Prefix caching: cache identical prompt prefixes to reduce repeat encoding cost for large context windows.
• Prompt compaction (auto‑compaction): summarize older context to reduce context rot and token cost while preserving signal.
• RAG / agentic RAG: selective retrieval rather than stuffing massive context windows.
• Mixed model strategies: use smaller/specialized models for parts of the task and larger models for final reasoning.
• Batching: allow non‑urgent workloads (e.g., evaluations) to run in batch for lower cost and better utilization.

Tool use and evaluating tool calls

• Use function‑calling benchmarks (e.g., Berkeley function calling leaderboard) to test tool invocation but also build multi‑turn, end‑to‑end agent tests.
• Consider preemptive heuristics to decide when to fetch external context (e.g., auto‑detecting pasted URLs or data that imply a tool call) to avoid an LLM call just to decide to call a tool.
• Treat tool orchestration as another component to be optimized (some routing decisions might not need full LLM power).

Engineering and infrastructure considerations

• Integrate agent orchestration into existing distributed inference infra; account for longer workflows and additional SLOs.
• Scheduling and compute utilization:
  • Host popular models to avoid wasted idle capacity.
  • Offer batch APIs for non‑urgent workloads to smooth demand and cut cost.
• Tradeoffs when agents take longer (minutes) vs. short real‑time agents (seconds) — different architectures and expectations.

Future directions for agents (guest’s view)

• Continued focus on efficiency: more compaction, better tooling, and faster inference to enable many short iterations.
• Convergence of evals toward production‑representative tasks so eval improvements translate to real gains.
• Developers will keep building novel agentic applications that seemed impractical months ago.

Actionable takeaways for builders

• Prioritize latency: fast models + fast tools = dramatically better UX and quality.
• Measure cost‑adjusted performance (intelligence/sec and intelligence/dollar), not just raw accuracy.
• Use reproducible, standardized eval harnesses; consider dynamic real‑time evals for search/novelty tasks.
• Start with minimal, efficient agent primitives (delegate, parallelize, compact context) before layering complex abstractions.
• Batch non‑urgent workloads to improve hardware utilization and pricing.

Notable quotes

• “The fastest agent in the race has the best evals.” (core theme: speed enables more iterations and higher quality)
• “If you have an agent that takes 5–7.5 seconds, you could do a few rounds. You can revise — and that generally ends up being higher quality.”

Resources & further reading (mentioned or relevant)

• Compound (agent platform) — discussed as Grok’s agent offering
• OpenBench — open‑source eval harness and repository referenced by the guest
• LangChain — example agent framework and high‑level abstraction tradeoffs
• Anthropic — “Building Effective Agents” (blog / research direction that influenced the discussion)
• Berkeley function‑calling leaderboard — function‑call/tool benchmarks
• Concepts: speculative decoding, prefix caching, prompt compaction, RAG / agentic RAG

If you want to implement or evaluate agents, prioritize fast inference and tool latency, run reproducible and production‑aligned evals, and measure not just accuracy but cost‑ and latency‑normalized performance.