Overview of Open to Debate: should we trust AI agents with our information?
This episode centers on a live debate about whether AI agents and assistants should be trusted with personal information, access to apps, and the ability to act on a user’s behalf. Meredith Whittaker of Signal argues that today’s agent systems are too risky and structurally insecure, while Neil Chilson of the Abundance Institute argues that agents can be useful, privacy-enhancing, and improved through real-world use. The discussion is framed around a key tension: convenience and productivity versus privacy, security, and control.
Core Debate Positions
Meredith Whittaker: do not trust agents as they exist today
- Agents often require deep permissions across contacts, calendars, messages, browsers, and payment methods.
- Their architecture can resemble “root access” to a user’s digital life, which she argues is dangerous by design.
- The biggest technical threat is prompt injection: malicious or misleading text can trick an agent into taking harmful actions.
- She argues that using agents can also expose other people’s data, not just the user’s own.
- Her broader concern is that agentic systems weaken application-level security and centralize power in major AI companies and OS vendors.
Neil Chilson: agents can improve privacy and user agency
- He argues that agents reverse the old internet bargain, letting people use tools to manage their own data for their own purposes.
- Agents can help people do things that were previously accessible only to wealthy users with human assistants.
- He says agents can also make privacy more practical by handling tedious tasks like:
- opting out of data brokers
- reading privacy policies
- using privacy-preserving tools
- He agrees that the technology is imperfect, but argues the answer is better design, better incentives, and continued use, not abandonment.
Key Topics Discussed
Privacy and data access
- How much information an agent needs to function well.
- Whether “delegating” tasks to software is meaningfully different from giving access to a human assistant.
- The risk that one person’s choice to use an agent affects other people’s privacy.
Security and prompt injection
- Meredith explains prompt injection as a core vulnerability in LLM-based agents:
- agents can’t reliably distinguish trusted instructions from malicious text
- this can lead to data leakage or unwanted actions
- Neil responds that security threats can be reduced through layered defenses:
- front-end filters
- permission controls
- model training
- better product design
Convenience vs. risk
- Neil emphasizes the real value of convenience at scale.
- Meredith counters that “convenience” can become a justification for deploying unsafe systems too quickly.
- The debate repeatedly returns to whether current systems are trustworthy enough for the access they request.
Market incentives and company responsibility
- Meredith argues companies have strong incentives to deploy risky products quickly.
- Neil argues companies are also constrained by liability, competition, and customer demand.
- Both agree that companies should be held accountable when harms occur.
Notable Examples Mentioned
- An agent booking a birthday party and needing access to:
- contacts
- calendar
- browser
- credit card
- messaging apps
- Reports of prompt injection affecting systems like:
- Microsoft Copilot
- Perplexity’s browser/agent tools
- Anthropic’s Chrome experiment
- A smart-home attack example where malicious calendar text caused unintended actions.
- An example of an AI agent exposing a user’s home address in a marketplace transaction.
- Neil’s example of using an agent to opt out of multiple data brokers with a short prompt.
Post-Debate Deeper Context
After the formal debate, John Donvan interviews Meredith and Neil about their backgrounds and reasoning:
Meredith’s perspective
- Her privacy concerns were shaped by her years at Google and her observation of the ad-driven surveillance business model.
- She drew parallels to historical abuses of data systems and mass surveillance.
- She sees encryption and privacy-preserving design as essential safeguards, not optional features.
Neil’s perspective
- He describes himself as optimistic because of complexity theory: large systems can’t be perfectly controlled, but they can be steered.
- He sees the internet and AI as systems that improve over time through iteration, not by freezing innovation.
- He believes the goal should be responsible deployment, not rejection.
Main Takeaways
- Meredith’s thesis: current AI agents are too invasive, too insecure, and too powerful to trust with sensitive information.
- Neil’s thesis: agents are imperfect but valuable tools that can expand access, improve convenience, and even help people protect their privacy.
- The debate is less about whether agents can do useful things and more about:
- how much access they should have
- who bears the risk
- whether the current architecture is safe enough
- A recurring theme is that trust must be earned through verification, not marketing claims.
Bottom Line
The episode presents a sharp but civil clash over the future of agentic AI. Meredith Whittaker argues that today’s systems are structurally unsafe and undermine privacy at the architectural level. Neil Chilson argues that agents can become practical tools for empowerment and privacy if developed and used responsibly. The debate ultimately asks listeners to decide whether the benefits of AI agents outweigh their risks right now.