Overview of Story: Inside Early Google — Race Conditions, Java Pain, and the Birth of AdWords
This episode (a Patreon bonus from Adam Gordon Bell's Co-Recursive) is an interview-based recollection by Ron Garrett about his year at early Google (circa 2000). Garrett describes how he joined to work on AdWords, his dislike of Java and JSP tooling, the human and cultural side of early Google, the first AdWords deployment (and first customer, “Lively Lobsters”), a serious billing outage caused by a C++ race condition writing garbage into MySQL, and his eventual return to JPL. The account mixes technical detail, startup culture anecdotes and management/engineering lessons.
Key events & timeline
- Ron Garrett discovered Google in 2000 and applied after being impressed by the search quality.
- He joined Google (under 100 employees at the time) to work on AdWords despite a pay cut and long weekly commute (Burbank ↔ San Jose flights).
- Assigned to implement AdWords in Java/JSP as the “Java evangelist,” though he disliked Java and preferred Lisp.
- Jeremy Chow (a colleague) wrote much of the production code; Garrett contributed architecture and the billing subsystem.
- AdWords first went live; first ad was from a company called Lively Lobsters.
- A major “billing disaster” occurred: bogus (very large) ad-served counts were written to the database, causing attempted/few successful large charges.
- Root cause: a race condition in a C++ ad server during shutdown—destructors freed memory that another destructor read and wrote to the DB, producing garbage numbers.
- Fixes: added sanity checks in the biller to prevent crazy charges, then ad-server bug fixed by its authors.
- After ~1 year Garrett left Google, returned to JPL with a promotion to principal; Google stock options later proved financially significant.
Technical highlights & problems
Architecture & components described
- Web UI (customers enter ads) → database → ad server (serves ads) → billing system (charges credit cards).
- Billing system designed with double-entry bookkeeping; Garrett implemented secure storage/encryption for credit cards to protect against dishonest employees.
Major technical incident — the billing disaster
- Symptom: extremely large ad-serve counts (millions/tens of millions) in MySQL where values should be reasonable hourly counts.
- Immediate consequence: declined charges for amounts above credit limits; but some charges below limits went through, requiring manual refunding and accounting fixes.
- Root cause: intermittent race condition in C++ ad server shutdown sequence. Two destructors interleaved:
- One destructor freed memory holding state.
- Another destructor read that memory and wrote the (now corrupted) values into MySQL.
- Why it took long: intermittent, triggered on ad-server restarts/shutdowns; corruption looked like random data, not DB engine failure.
- Short-term mitigation: add sanity checks in billing to reject implausible values and alert operators.
- Long-term fix: ad-server code corrected to remove the race and eliminate corrupted writes.
Tooling pain
- Garrett found JSP development painful: JS P pages mixing HTML+Java, no syntax-aware editor or delimiter balancing, inscrutable compiler/error messages far from the true error location. This slowed debugging dramatically.
- He emphasizes the value of tooling, syntax highlighting and editors for complex template-based server pages.
Cultural & personal observations
- Google culture: incredibly smart colleagues (many C++/Perl experts); a “move fast and break things” mindset where finding/fixing problems was expected over blame.
- Garrett felt isolated initially (shared office with another new hire, separated from open cubicles) which hindered cultural integration.
- He describes imposter syndrome — feeling in the bottom quartile among peers despite previous seniority.
- Startup perks/oddities: staying in a room at Susan Wojcicki’s house, Sergey Brin’s unicycle as a decompression tool, and general Silicon Valley eccentricities.
- Management decisions: Larry Page favored no pre-review of ads (ads go live immediately with post-facto moderation). Garrett worried about legal exposure but Page prioritized instant customer gratification — a decision Garrett later viewed as correct for AdWords’ success.
- Personal costs: exhausting commute, family stress, working late and constant pressure to hit a three-month delivery timeline.
Notable quotes / memorable lines
- On Java: “It was designed to turn programmers into cogs and wheels… the amount of boilerplate drove me bananas.”
- On early Google talent: “I felt like I was in the bottom 25%, maybe the bottom 10%… I was completely unaware of the scope of things that I was unaware of.”
- On product philosophy: Larry Page: “We’ll worry about that if it happens” — in reference to legal or offensive ads; choice to make ads live immediately.
Lessons & recommendations (actionable takeaways)
- Managers: don’t force someone to both learn a new stack and deliver a production system on a tight timeline. Separate learning time from delivery responsibilities.
- Build sanity checks/validators for critical financial flows. Never trust unvalidated external data (or inter-service counters) when charging money.
- Race conditions in native code (C/C++) can produce subtle corruption that looks like DB garbage — ensure correct shutdown ordering, memory ownership, and thread-safety; test shutdown/restart paths under load.
- Provide proper tooling (syntax-aware editors, linters) for template-heavy stacks (e.g., JSP, PHP, mixed HTML+server code) — saves huge debugging time.
- Startup culture: “move fast and fix” requires good incident response processes and the expectation that teams will repair damage without punitive blame.
- Product strategy: instant feedback/gratification for users can be a decisive UX advantage (as with Google’s decision to allow ads to go live immediately).
Short summary / “in one paragraph”
Ron Garrett recounts joining early Google to build AdWords under a tight three-month deadline, grappling with a technology choice he disliked (Java/JSP), cultural dislocation, and heavy travel. He implemented the billing system (including secure card storage), launched the first live ads (first customer: Lively Lobsters), then battled a serious billing outage caused by a C++ race condition that wrote corrupted counts into MySQL. The incident led to adding sanity checks and fixing the ad-server bug. Garrett reflects on organizational decisions (instant ad publication), the brilliance of colleagues, startup stress, and how a brief stint at Google ultimately changed his life financially and professionally.
Who should read/listen
- Engineers building online billing/ad platforms (race conditions, validation).
- Managers in charge of staffing and timelines (learning vs shipping).
- Anyone interested in early Silicon Valley/Google history and product decisions behind AdWords.