Overview of Distributed Databases with Peter Mattis
This episode features Peter Mattis, co-founder and CTO of Cockroach Labs, in a wide-ranging conversation about distributed systems, database internals, and how AI has transformed his coding workflow. Peter traces his path from building early graphics software like GIMP, to working on Gmail’s backend at Google, to helping build Colossus and eventually CockroachDB. A central theme is that strong fundamentals—especially around data structures like B-trees, consistency models, and storage architecture—still matter deeply, even as AI dramatically increases engineering velocity.
Peter Mattis’ Path Into Tech
Peter got into computers early, helped along by a home environment full of Apple machines and an interest in gaming and experimentation. In college, he started in mechanical engineering, but quickly realized CS was a far better fit after comparing the workload and seeing how much more natural programming felt.
GIMP and Early Open Source Impact
- At college, Peter and his roommate built what became the GIMP while trying to make something like Photoshop.
- He also worked on GTK, a graphics library that grew into a major part of the Linux desktop ecosystem.
- The experience taught him two lasting lessons:
- Big software projects often begin naively.
- You should not be discouraged if others appear to be working on the same idea.
Google, Gmail, and the Power of Timing
- Peter was recruited to Google early on, initially said no because of the commute, then joined in 2002.
- He worked on the backend threading, message storage, and indexing for what became Gmail.
- Gmail’s launch was a major shock to the email industry because it offered:
- far more storage than competitors,
- fast search,
- and a free model that initially seemed impossible.
- He noted that B-trees were used in Gmail’s backend, especially for message threading and unread counts.
Lessons From Google Systems Work
Build Systems: From Makefiles to Bazel
- Peter helped kick off Google’s next-gen build system, which started as an attempt to replace huge, hard-to-maintain makefiles.
- The goal was to express dependencies at a higher level and compile them into something more efficient.
- This evolved into Blaze, later Bazel externally.
Colossus: Google’s Successor to GFS
- Peter also worked on Colossus, the successor to Google File System (GFS).
- Key ideas:
- scalable distributed storage for huge datasets,
- metadata stored in Bigtable,
- bootstrapping through practical hacks before fully cleaning up the architecture,
- use of erasure coding with Reed-Solomon to reduce storage overhead compared to simple replication.
- He emphasized that storage systems are often constrained by hardware realities:
- HDDs vs SSDs,
- network latency,
- and even the speed of light over long distances.
Why Latency Matters So Much
- Peter repeatedly stressed that performance engineering must account for real hardware limits.
- Distributed systems often live at the edge of:
- disk latency,
- network round-trip times,
- and memory/cache behavior.
- The main lesson: the more you understand the machine, the more efficient your software can become.
CockroachDB: Distributed Database Design
Why Peter Cofounded Cockroach Labs
- Peter, along with former colleagues, had previously explored building a database while at a startup.
- They wanted a database that could handle failure gracefully and reduce the burden on application developers.
- That became CockroachDB, designed for resilience and horizontal scale.
Automatic Sharding and Range-Based Partitioning
- Peter contrasted:
- manual sharding, where application developers must split and manage data themselves,
- with automatic sharding, where the database manages distribution.
- CockroachDB uses range-based partitioning, which is conceptually similar to a B-tree:
- contiguous key ranges are stored and indexed,
- the system manages which node owns which range.
- This reduces operational pain and avoids making developers act like database engineers.
Strong Consistency and Raft
- CockroachDB aims for strong consistency and transactional guarantees.
- Peter explained:
- atomicity: all-or-nothing transactions,
- isolation/serializability: concurrent transactions should appear ordered,
- eventual consistency: faster, but with stale reads and more application complexity.
- For consensus, CockroachDB uses Raft:
- typically 3 replicas,
- writes are replicated to a quorum,
- more replicas can improve durability at the cost of latency.
- He emphasized that these trade-offs become especially important in multi-region deployments.
Mission-Critical Use Cases
- CockroachDB is used for systems that cannot easily go down:
- financial/trading systems,
- shopping carts,
- infrastructure tied to revenue or customer trust,
- services that need to survive node, rack, or even region failures.
- Peter framed the company’s value proposition as removing the burden of distributed systems from application teams.
Data Structures and Performance: B-Trees Everywhere
Why B-Trees Keep Coming Up
Peter repeatedly returned to B-trees as a foundational structure in databases and storage systems.
Where They Show Up
- Gmail backend threading/indexing
- database indexes
- internal Google systems
- data structures for maps and storage metadata
Why They Matter
- B-trees improve cache locality compared to pointer-heavy trees like red-black trees.
- They store multiple keys per node, reducing pointer chasing and memory overhead.
- Peter described how practical systems work often starts with noticing inefficiencies in existing libraries and asking whether a different structure can fit the access pattern better.
Better Maps and Swiss Tables
- Peter also discussed contributing to a faster map implementation inspired by Swiss tables.
- His work helped improve Go’s map performance in some cases.
- The broader lesson: pay attention to profiles, memory use, and cache behavior—there are often wins hidden in “standard” data structures.
AI and the Return to Hands-On Coding
From Slowing Down to Coding More Than Ever
- Peter wrote enormous amounts of production code pre-AI, at one point reaching around 100,000 lines of code per year.
- He stepped back from day-to-day coding between roughly 2022 and 2024 due to leadership responsibilities.
- AI coding tools changed that:
- he returned to coding,
- and found himself more productive than before.
How He Uses AI Tools
- He moved from autocomplete tools to more agentic workflows.
- Tools mentioned included:
- Claude Code,
- Codex,
- desktop apps and multi-agent setups.
- He often runs multiple sessions and sub-agents in parallel, using AI as both:
- a coding assistant,
- and a design partner or “sparring partner.”
AI Is Good, But Not Self-Directed
Peter was clear that AI models are powerful but imperfect:
- they can be lazy about testing,
- they may miss performance concerns,
- they still need strong human guidance.
His fix is not to avoid AI, but to tighten guardrails:
- comprehensive tests,
- property-based testing,
- metamorphic testing,
- deterministic simulation,
- security review,
- performance checks.
Code Review Is Changing
- He believes code review remains important, but the nature of it is shifting.
- Human reviewers may soon inspect code less directly, with AI doing more of the first-pass validation.
- He expects a future where humans define constraints and review higher-level behavior, while AI handles more of the mechanical verification.
Main Takeaways
1. Fundamentals still matter
- B-trees, consistency, caching, and storage layout are not obsolete.
- They are still central to high-performance systems.
2. Distributed systems are about trade-offs
- Strong consistency, replication, and low latency are difficult to combine.
- Good systems engineering is about making those trade-offs explicit and manageable.
3. AI amplifies expertise
- Peter’s strongest point was that AI doesn’t replace deep engineering knowledge.
- It makes good engineers much more effective, especially when they know what to ask and how to verify.
4. Testing and rigor matter even more with AI
- More output means more chances for bugs.
- The solution is not less AI, but better guardrails and stronger discipline.
5. The best engineers stay curious
- Peter’s career shows a pattern of constantly digging into adjacent systems:
- graphics,
- search,
- build systems,
- storage,
- databases,
- AI tooling.
- That curiosity is a big part of why he has remained effective across decades.
Closing Thought
Peter’s story is a reminder that strong systems thinking compounds over time. He built foundational software long before AI, and now he’s using AI to extend that same depth of engineering into a new era. His message is clear: if you know the system well, AI can help you move faster, think bigger, and build better.