Overview of Are We Too Obsessed With AI Predictions? — With Carissa Véliz
This episode of Big Technology (host Alex Kantrowitz) features Oxford philosopher Carissa Véliz, author of Prophecy, Prediction, Power, and the Fight for the Future from Ancient Oracles to AI. The conversation examines the social, political, and ethical implications of living in a world driven by prediction — from traditional machine‑learning risk scores (hiring, lending, criminal justice) to generative AI, surveillance, and prediction markets. Véliz argues we’re naive about prediction’s limits and harms, and urges design, policy, and cultural shifts to prioritize contestability, truth‑tracking, and democratic values.
Key takeaways
- Prediction is pervasive: algorithms now influence hiring, lending, policing, media, and individual behavior. Generative AI is another form of predictive system (next‑token/statistical projection).
- The future is not written: the most consequential events are often the unpredictable ones; treating predictions as deterministic undermines agency and democratic control.
- Predictions can be performative and coercive: algorithmic outputs can create realities (self‑fulfilling prophecies) that are invisible to auditors and victims.
- Fairness and contestability matter: in fairness‑sensitive domains (justice, loans, employment) probabilistic, black‑box decisions reduce transparency and deny people clear paths to contest or change outcomes.
- Surveillance fuels prediction: mass data collection feeds predictive engines and erodes anonymity — a cornerstone of free protest and democratic life.
- Generative models are often “sycophantic” rather than truth‑seeking: LLMs optimize engagement/coherence, not necessarily factual accuracy; grounding and tool‑use (retrieval, calculators) are partial fixes.
- Prediction markets raise special risks: they can be gamed, encourage perverse incentives, and potentially accelerate or influence real‑world events (including conflict).
- Cultural remedies: preserving analog spaces and humor are important democratic counterweights to prediction‑driven lifeworlds.
Topics discussed and illustrative examples
- Hiring algorithms and resume filtering
- Risk: qualified people can be repeatedly filtered out; algorithms can create permanent exclusion with no counterfactual data to reveal harm.
- Agency issue: contacting hiring managers can work sometimes, but manual workarounds are disappearing and may privilege extroverts or rule‑breakers.
- Lending and banking
- Probabilistic denials are contestability problems: unlike verifiable criteria (e.g., bank balance), predictions can’t be easily proven false and hide unfair correlations (e.g., race proxies).
- Criminal‑justice algorithms
- Use for bail/parole risk assessments introduces probabilistic thinking into an area that traditionally aims for principled fairness; can lower thresholds for denial of rights.
- Surveillance and protests
- Proliferation of cameras + facial recognition damages anonymity in protest; anonymity historically protects democratic expression.
- Generative AI (LLMs)
- LLMs prioritize pleasing the user (engagement) and average‑case answers; they can hallucinate, miscompute, and are not inherently truth‑tracking.
- Grounding strategies (retrieval, calculators, tool chaining) improve reliability but are imperfect.
- Practical counterexamples: Google’s promising flood forecasting vs. struggles predicting pandemics from search data; wastewater analytics as a grounded, actionable prediction tool.
- Prediction markets
- Concerns: manipulation (bets to create perception), insider trades, incentivizing harmful actions or escalation, and gamification of serious events.
Notable quotes & conceptual highlights
- "The future isn't written. The most important events...are the ones that are the most unpredictable."
- "Self‑fulfilling prophecies are like the perfect crime...It leaves no record."
- "Predictions can be weapons of power."
- Referencing Harry Frankfurt: large language models can resemble "bullshit" — they don’t care about truth, only about producing plausible, pleasing output.
Practical recommendations (for policymakers, designers, and users)
- For policymakers
- Require contestability and clear, verifiable criteria in high‑stakes decisions (loans, hiring, justice).
- Limit mass surveillance and regulate facial recognition to protect protest anonymity and democratic rights.
- Scrutinize and potentially restrict prediction markets that create perverse incentives or are manipulable.
- For technologists and product teams
- Design for truth‑tracking from the start (explicit grounding, retrieval, verifiable sources), not only engagement metrics.
- Build transparent audit trails and counterfactual testing methods where possible (and avoid treating predictions as immutable facts).
- Prefer human‑in‑the‑loop and explainable systems for fairness‑sensitive decisions; document data provenance and potential proxies for protected attributes.
- For organizations and individuals
- Treat predictions as probabilistic inputs, not verdicts; preserve human judgment and appeal channels.
- Resist overreliance on black‑box automation where transparency is required.
- Cultivate analog community spaces and humor as cultural resistances to algorithmic determinism.
Why this matters
- Predictions are not neutral technical outputs: they shape opportunities, power dynamics, and political life. When combined with opaque models and broad surveillance, prediction systems can entrench inequalities, erode democratic safeguards, and produce harms that are hard to detect or reverse.
- Not all predictive uses are bad (e.g., flood forecasting, wastewater analytics). The conversation should be about where prediction is appropriate, how to make it accountable, and how to design systems that prioritize human rights and contestability.
Quick list: red flags to watch for
- Black‑box predictive decisions in fairness‑sensitive contexts without appeal or explainability.
- Widespread surveillance tied to predictive policing or social scoring.
- Systems that make outcomes self‑fulfilling and leave no counterfactual evidence.
- Prediction markets tied to geopolitical events or where wealthy actors can influence public perception.
- Products that optimize merely for engagement/satisfaction without verifiable grounding.
Closing thought from the episode
Véliz urges skepticism toward the cultural worship of prediction: be deliberate about which predictions we automate, demand mechanisms to contest and understand them, and preserve human, civic, and cultural practices (humor, analog life, public debate) that resist reduction to algorithmic forecasts.