This Is How to Tell if Writing Was Made by AI

Summary of This Is How to Tell if Writing Was Made by AI

by Bloomberg

48m•April 2, 2026
Listen to Episode

Overview of Odd Lots — "This Is How to Tell if Writing Was Made by AI"

This episode of Bloomberg's Odd Lots (hosts Joe Weisenthal and Tracy Alloway) interviews Max Spiro, founder & CEO of Pangram Labs, about detecting AI-written text. They discuss why detection matters, how Pangram’s detector works, its measured performance, examples of use (platform moderation, teachers, journalists, consumers), broader societal implications (trust, provenance, norms), and technical limits and adversarial challenges.

Key takeaways

  • Pangram Labs builds ML models to classify text as human-written, AI-generated, or AI-edited (assisted).
  • Pangram reports ~1 in 10,000 false positives (labeling human text as AI) and ~1% false negatives (missing AI text) in typical conditions.
  • Pangram’s approach goes beyond simple metrics (perplexity/burstiness) and uses large deep models trained on millions of paired human/AI examples and active learning.
  • Rough prevalence estimates: Pangram suggests ~40% of the indexed internet content is AI-generated; >50% of new Medium articles (as of ~1.5 years ago) were AI-generated; Reddit rose from ~7% to ~10% AI content.
  • Major concerns: erosion of heuristics that signaled "serious" writing (good grammar/punctuation), reputational risk if detectors make false claims, platform incentives that can encourage AI content, and the need for provenance systems for media (C2PA, hardware-based proofs).

How Pangram Labs’ detector works

  • Human baseline: Pangram first tested how well humans can guess AI vs. human writing (found ~90% achievable for a skilled evaluator).
  • Paired training data: For many human examples Pangram generates close AI “mirrors” (same topic/length/style) so the model learns contrastive differences.
  • Deep learning classifier: Trains large neural networks (initially BERT, later much larger models) that output an AI-vs-human prediction rather than next-token prediction.
  • Active learning loop: Scan large corpora, find model errors (false positives/negatives), add those borderline examples back to training to improve performance.
  • AI-edit detection: Pangram generates AI “editing” variants (e.g., “clean up grammar”) of human text and measures embedding-space distances to estimate light/moderate/heavy AI assistance.
  • Model interpretability: Embedding clusters can reveal which frontier model family (e.g., Claude, GPT variants, multilingual models) likely produced text, even if Pangram isn't trained to label model names explicitly.

Performance and limits

  • Reported accuracy: ~99%+ detection for straightforward AI outputs; ~1 in 10,000 human false-positive rate on their large human corpus.
  • Adversarial robustness: Pangram handled aggressive obfuscation experiments (multiple machine translations across languages) in tests, but adversarial prompting remains a risk and can raise false negatives.
  • Scaling: As LLMs become more complex, Pangram increased model capacity (parameter counts) to capture richer output distributions. Detection must continue evolving to keep pace with generation improvements.
  • Not a silver bullet: No detector is perfect; false positives/negatives will occur, especially near the human/AI boundary and with intentionally evasive prompts.

Why detection matters (use cases & harms)

  • Platform moderation: Quora and other platforms use detectors to surface and remove AI-driven spam or bot content.
  • Academic integrity & teaching: Educators need reliable tools to tell student-produced work from AI-generated submissions.
  • Journalistic integrity & reputation: Newsrooms and audiences care whether reported copy is original or AI-assisted; undisclosed AI use can be reputationally damaging.
  • Consumer protection: Detecting fake reviews, SEO-generated “slop,” or coordinated bot campaigns (e.g., brand-promotion on Reddit) helps preserve signal quality.
  • Societal risk: Large-scale flooding of low-effort AI content (“AI slop”) can erode trust, degrade the quality of search results, and make genuine human voices harder to find.

Notable stats & examples from the episode

  • Pangram estimate: ~40% of the internet is AI-generated (driven largely by SEO and low-cost content mills).
  • Medium: Over 50% of newly written Medium articles were AI-generated in a study referenced.
  • Reddit: Pangram saw ~7% AI a year ago, now a little over 10%.
  • Example: A Guardian writer’s Winter Olympics article was flagged by Pangram; analysis showed the author had increased AI usage in 2024.

Broader implications, norms & technical complements

  • Norms: Max and hosts emphasize social norms—disclosure of AI assistance—are important. Acceptable use (e.g., grammar fixes) vs. undisclosed generation should be defined.
  • Provenance: For images/video, industry efforts (C2PA) aim to embed device/hardware provenance to certify authenticity rather than simply labeling outputs as “AI.”
  • Platform incentives: Big tech has mixed incentives—product teams push generative features, while search/product quality teams must fight AI slop in results.
  • The “craft” signal is severed: Historically, polished prose implied seriousness; LLMs replicate polish, so heuristics must change.

Risks & ethical concerns

  • Reputation risk: False positives could falsely accuse creators of using AI and harm careers.
  • Arms race: Generators can be tuned to evade detectors; detectors must scale and adapt continually.
  • Biases: Early detection metrics (perplexity) produced false positives for non-native English text; advanced detectors try to avoid such bias but risks remain.
  • Data provenance for training: As more text is AI-generated, ensuring quality human training data (pre-2023 reservoir, trusted actors) becomes critical.

Practical recommendations / action items

For individual readers

  • Use detectors as informative signals, not final judgments. Treat results as part of a broader evidence set.
  • Prefer transparent disclosure: if you use AI to assist, label it—lighter assistance (grammar, editing) vs. generative contributions.
  • Be skeptical of reviews/articles that look formulaic—run suspicious items through detectors if important (e.g., product choices).

For educators & employers

  • Combine technical detection with assessment design—assignments that require process artifacts, drafts, or in-person components.
  • Update honor codes and policies to define acceptable AI-assistance and penalties for undisclosed use.

For platforms & regulators

  • Invest in provenance systems (especially for audio/video/image) and authenticated content channels.
  • Encourage norms for disclosure and build friction for undisclosed mass-generated content (rate limits, verified posters).
  • Support independent audits of detector systems to limit false-positive harm and bias.

For developers & researchers

  • Continue adversarial testing (prompt obfuscation, translation cascades) and scale detectors to handle richer LLM output distributions.
  • Publish evaluation datasets and error rates to increase transparency about limits and edge cases.

Notable quotes (condensed)

  • Max Spiro: “If the vast majority of these decisions line up with how the frontier models are doing it, then it’s vanishingly unlikely that this was written by a human.”
  • Hosts’ framing: The web may already be “40% AI slop,” meaning increasingly the link between polished prose and human craft is severed.

Bottom line

Detecting AI-written text is tractable and improving: Pangram demonstrates high accuracy via large-scale contrastive training and active learning. But detection is not infallible, adversaries can try to evade it, and detection alone won’t solve broader problems of provenance, incentives, and norms. Addressing “AI slop” requires a mix of technical defenses, platform policy, provenance standards, and cultural norms around disclosure.