Summary of The good, the bad, and the AI apps

Overview of The good, the bad, and the AI apps

In this episode of the Stack Overflow Podcast, host Ryan Donovan speaks with Benny Chen, co-founder of Fireworks AI, about one of the hardest problems in AI product development: defining what “good” actually means for an AI application. The conversation focuses on evaluation strategy at scale—how teams convert messy user feedback into measurable signals, how to separate deterministic safety checks from subjective product quality, and how inference infrastructure is changing as AI systems become more agentic and automated.

Main Themes and Takeaways

Why “good” is hard to define in AI

• AI applications are inherently non-deterministic, so quality is harder to measure than in traditional software.
• The real goal is not just model accuracy, but making the customer’s customer happy.
• “Good” often needs to be translated into:
  • repeatable signals,
  • actionable metrics,
  • and measurable improvements over time.

Turning sparse feedback into evals

• Benny describes user feedback as often “sparse”:
  • app reviews,
  • thumbs up/down,
  • Twitter complaints,
  • qualitative reports.
• Fireworks helps customers convert those signals into structured evals that can be used for:
  • model improvement,
  • fine-tuning,
  • model selection,
  • and release validation.
• A key idea: social/media complaints should be turned into reproducible test cases rather than treated as anecdotes.

A practical framework for building evals

Benny breaks the process into a few steps:

1. Decompose the complaint

   • Identify the specific symptom the model is failing on.
   • Problems are usually broader than one issue, but you need a clear repro to start.

2. Use language models as judges

   • A model can often determine whether a specific output is correct or not.
   • Example: if an HTML tag is wrong, the eval can ask whether that tag is incorrect.

3. Close loopholes iteratively

   • Models will often find the easiest way to satisfy the eval rather than the real user need.
   • Teams need to keep refining evals as they discover failure modes.

4. Separate safety from product quality

   • Safety issues like prompt leakage or unsafe responses should be deterministic CI tests.
   • Product quality issues often require scoring and trade-offs rather than strict pass/fail rules.

Deterministic tests vs. scored evals

• Deterministic evals are best for things that must never fail:
  • prompt leaks,
  • safety violations,
  • policy breaches.
• Scored evals are better for subjective trade-offs:
  • reasoning quality,
  • code generation,
  • content quality,
  • spatial reasoning.
• Benny stresses that deciding how to weight metrics is a product decision, not something to leave entirely to the model.

Product teams, leadership, and metric weighting

• The conversation highlights how metric choices often come from leadership priorities.
• Example from Meta: increasing the weight of comments vs. likes was ultimately a strategic decision.
• Benny argues that organizations need to be explicit about why they value one metric over another, and by how much.

Closed-loop feedback and long-term validation

• Proxy metrics can help, but they are not always enough.
• Teams should validate eval weights against long-term outcomes when possible.
• In practice, the challenge is that:
  • some effects are hard to measure,
  • some feedback arrives too late,
  • and some “badness” only shows up in broader user behavior or public reaction.

Fireworks AI’s Scale and Infrastructure Focus

Massive inference workloads

• Benny notes that Fireworks processes enormous volumes of tokens per day and positions the company as a major inference provider.
• Their work spans:
  • customer deployments,
  • internal evals,
  • performance optimization,
  • and infrastructure reliability.

Key infrastructure optimizations

• They focus on improving:
  • speculative decoding,
  • model training for drafters,
  • hardware-aware optimization,
  • quantization across hardware types,
  • scheduling and failover,
  • and overall throughput under high utilization.

Agentic systems change usage patterns

• As AI becomes more capable, teams are using it for more of their day-to-day work:
  • debugging,
  • finance,
  • design,
  • EDA,
  • internal operations.
• This increases token usage overall, even as per-token costs fall.

Open Source, Verticality, and Reuse

Open source is central

• Fireworks emphasizes open source as a way to:
  • share tooling,
  • collaborate with the community,
  • and give customers more control over their stack.
• Their eval protocol is designed to be reusable:
  • in GitHub Actions,
  • in PyTest,
  • and in reinforcement learning rollouts.

Evals are often vertical-specific

• Benny is skeptical that one eval framework or one set of metrics will generalize across industries.
• Examples:
  • coding,
  • legal due diligence,
  • design.
• Even if some patterns overlap, the evaluation criteria and failure modes differ significantly by domain.

The Future: Online Training and Data Flywheels

The next big opportunity

• Benny is especially interested in online SFT / on-policy training using traces from production.
• This mirrors how recommendation systems have long used real-world feedback loops.
• He sees this as a promising way to build a data flywheel for AI apps, though it remains technically tricky.

Why it matters

• Better production feedback loops could help customers:
  • improve models continuously,
  • adapt faster to real user behavior,
  • and reduce manual labeling overhead.
• But doing this correctly requires careful handling of:
  • data quality,
  • evaluation design,
  • and model drift.

Notable Insights

• “The model will always find the laziest way to make you happy.”

  • A reminder that evals must be designed carefully or models will game them.

• Metrics are a product decision.

  • Weighting comments over likes, or one quality dimension over another, should be intentional.

• Use the same evals across the lifecycle.

  • Good evals should support testing, tuning, and reinforcement learning—not just one-off validation.

• There’s no clean answer to reward shaping organizations.

  • Internal incentives, product priorities, and customer outcomes all interact in messy ways.

Outro and Community Notes

• The episode closes with:
  • a Stack Overflow badge shoutout for a highly upvoted answer,
  • Ryan Donovan’s contact details,
  • and Benny’s invitation for listeners to connect with Fireworks AI via fireworks.ai.
• Benny also mentions they’re looking for strong candidates and want to work with the “best and brightest.”

Bottom Line

This episode is a practical look at how AI companies should think about quality at scale. The core message is that AI evals are not just a technical exercise—they’re a mix of product judgment, infrastructure discipline, and iterative feedback design. To build useful AI apps, teams need to turn messy real-world signals into structured, reusable evaluation systems that can evolve with the model.