Overview of Why PMs are suddenly valuable again | Tulsee Doshi (Head of Product, Gemini)
In this episode, Lenny Rachitsky talks with Tulsee Doshi, Head of Product for Google’s Frontier Models, about how AI model development changes the role of product management, why PMs are becoming more important again, how Google thinks about competing at the frontier, and what it takes to ship AI products responsibly. Tulsee shares practical advice on evaluating models, building with uncertainty, designing UX for different user types, and keeping teams effective in a fast-moving AI era.
Key Themes and Takeaways
Models are not fully predictable
Tulsee emphasizes that model development is part science, part “magic.” Even with strong training goals and evals, models can surprise teams in ways that create new product opportunities.
- PMs should not only define desired outcomes, but also study what the model actually becomes.
- The job is to identify where a model shines, what feels unique, and how to turn those strengths into user value.
- She compares model training to “Snape’s potions class”: you can control the ingredients, but the outcome still has surprise in it.
Google wants to compete at the frontier
Tulsee pushes back on the idea that Google is merely trailing OpenAI and Anthropic. She says Google is still aiming to win at the frontier while also building for scale across its full product ecosystem.
- Google’s strategy is “full stack,” from hardware to foundational models to end-user products.
- The company is also focused on serving multiple segments: consumers, developers, enterprises, and security teams.
- Frontier performance matters, but so does building efficient models that power products used by billions.
Safety is embedded in the development process
Because Tulsee previously led product for Google’s Responsible AI team, she brings a systems-level view of AI safety.
- Safety is not treated as a post-launch add-on; it is built into training and deployment.
- Monitoring model behavior, including chain-of-thought alignment checks, helps catch misalignment before deployment.
- She also argues AI can strengthen safety by helping defenders find and patch vulnerabilities faster.
PMs Are “Back” in AI
Why PMs are becoming more valuable
Tulsee agrees with the current wave of sentiment that strong PMs matter more now that it is easier to build software.
- When building is cheap and fast, taste becomes more valuable.
- PMs help distinguish signal from noise and decide what is actually worth building.
- They also play a bigger role in aligning teams, creating structure, and turning speed into coherent product direction.
What Google looks for in PM hires
Tulsee says the strongest PMs for AI are not just organized or strategic — they have developed intuition for models.
- They “understand the vibes” of models by using them every day.
- They should have opinions, stress test models, and be willing to get hands-on with data and evals.
- Adaptability is critical because roadmaps can change quickly as models improve every few weeks.
Practical Advice on Evals and Model Understanding
How to get better at evals
Tulsee recommends starting small instead of trying to build a huge benchmark right away.
- Begin with just 5–20 prompts.
- Use your own judgment as the rater before trying to scale.
- Focus first on understanding what “good” looks like for a specific use case, then expand from there.
How to develop model intuition
She suggests testing models on tasks you genuinely care about.
Examples she gave:
- Writing and editing
- Creative generation
- Financial analysis
- 3D or visual tasks
- Synthesizing information
The key idea: if the task matters to you, you’ll notice subtle improvements and failures more clearly.
Important lesson: evals must be holistic
A good eval is not just about the raw model.
- Judge quality matters.
- Serving stack and mitigations affect results.
- Audio quality, tone, and naturalness can change depending on deployment details.
- You need an end-to-end view of what the user actually experiences.
UX, Product Design, and AI Complexity
Different users need different AI interfaces
Tulsee argues there is no single “right” UX for AI.
- For broad consumer products like Search, simplicity and accessibility matter.
- For power-user tools, more flexible and complex interfaces can be appropriate.
- Google is experimenting with new paradigms like NotebookLM, Flow, Genie, Veo, and other form factors.
The model itself has a UX
One of her more interesting points is that model behavior is part of the user experience.
- How transparent the model is
- How it communicates
- What follow-up questions it asks
- How it handles ambiguity
These are all UX decisions, even if they happen inside the model rather than the product shell.
Building Ahead of the Model: The 60-20-20 Rule
Tulsee introduces a practical framework for building with rapidly improving AI:
- 60%: features that work now
- 20%: features that are imperfect but usable and likely to improve soon
- 20%: features where the model only shows “glimmers of light,” but the opportunity is compelling
Why this matters
Instead of only shipping what is fully reliable today, teams should build with the future in mind.
- The model will likely be better soon.
- Products should be flexible enough to improve with it.
- Rigid systems can become bottlenecks if they are over-optimized for older model behavior.
Team Development and Avoiding Burnout
How Tulsee helps her team upskill
She says her team is learning from each other through shared tools and workshops.
- PMs who build useful internal workflows share them with others.
- Workshops help spread skills around data analysis, model evaluation, and AI tooling.
- Google is even planning dedicated time away from normal work to experiment and build.
Burnout is not just about hours
Tulsee argues burnout often comes from two things:
- Chaos and churn
- Work that feels pointless or purely grunt work
Her view is that AI can reduce burnout if it removes repetitive work and frees people for more creative, strategic tasks.
Managing AI Slop and the New PM Workflow
AI should generate data, not insight
Tulsee draws a clear line between what AI should do in docs and what humans should do.
- AI is great for pulling metrics, trends, and usage summaries.
- Humans should provide the interpretation, judgment, and product insight.
- This helps avoid generic, low-value AI-written docs.
How AI changes the PM role
AI has already transformed:
- Prototyping
- Data analysis
- Faster decision-making
- The speed of iteration
But the bigger shift, Tulsee says, is that PMs now need to go beyond writing requirements — they need to bring real substance to the work.
Notable Stories and Context
Equity and representation in model evals
Tulsee shared a strong example from her Responsible AI work:
- Google replaced the outdated Fitzpatrick skin-tone scale with a more representative 10-tone scale developed by Professor Monk.
- That change improved evaluation quality for image-related products and helped make outputs more inclusive.
- It shows how eval design can materially improve product fairness.
Research that maps to today’s agent world
She also discussed early Stanford research on “flash teams” — a precursor to modern agent swarms.
- The original work was about organizing human crowdworkers into efficient task teams.
- Now similar principles are relevant for agent systems that can coordinate on their own.
- Her point: many current AI trends have roots in earlier human collaboration research.
Lightning Round Highlights
Books Tulsee recommends
- Exhalation by Ted Chiang
- Creativity, Inc. by Ed Catmull
- The A Court of Thorns and Roses / Throne of Glass series for fun reading
Favorite show
- Severance
Favorite AI product besides Gemini
- WhisperFlow, which she uses because of wrist issues and its excellent speech-to-text experience
Personal motto
- “Everything happens for a reason”
- “Seek serendipity” — put yourself in positions where luck can happen
Final Takeaways
- AI product development is still partly unpredictable, so PMs matter more, not less.
- Great PMs in AI need taste, technical curiosity, adaptability, and a strong feel for model behavior.
- Evals should be simple, practical, and tied to real user tasks.
- The best AI products will balance immediate utility with room to grow as models improve.
- Safety, UX, and strategy all need to be designed together from the start.
Actionable Advice from the Episode
- Start evals with a small set of prompts before scaling up.
- Test models on tasks you genuinely care about.
- Don’t build rigid systems that assume today’s model behavior will stay constant.
- Treat AI-generated docs as data sources, not sources of insight.
- Give teams dedicated time to learn tools, experiment, and share useful workflows.
- Look for PMs who actively use models and can explain what feels better or worse about them.