How Google Tests Software, 1st edition

Foreword by Alberto Savoia     xiii
Foreword by Patrick Copeland     xvii
Preface     xxiii

Chapter 1: Introduction to Google Software Testing     1
Quality?Test     5
Roles     6
Organizational Structure     8
Crawl, Walk, Run     10
Types of Tests     12

Chapter 2: The Software Engineer in Test     15
The Life of an SET     17
Development and Test Workflow     17
Who Are These SETs Anyway?     22
The Early Phase of a Project     22
Team Structure     24
Design Docs     25
Interfaces and Protocols     27
Automation Planning     28
Testability     29
SET Workflow: An Example     32
Test Execution     40
Test Size Definitions     41
Use of Test Sizes in Shared Infrastructure     44
Benefits of Test Sizes     46
Test Runtime Requirements     48
Case 1: Change in Common Library     52
Test Certified     54
An Interview with the Founders of the Test
Certified Program     57
Interviewing SETs     62
An Interview with Tool Developer Ted Mao     68
An Interview with Web Driver Creator Simon Stewart     70

Chapter 3: The Test Engineer     75
A User-Facing Test Role     75
The Life of a TE     76
Test Planning     79
Risk     97
Life of a Test Case     108
Life of a Bug     113
Recruiting TEs     127
Test Leadership at Google     134
Maintenance Mode Testing     137
Quality Bots Experiment     141
BITE Experiment     153
Google Test Analytics     163
Free Testing Workflow     169
External Vendors     173
An Interview with Google Docs TE Lindsay Webster     175
An Interview with YouTube TE Apple Chow     181

Chapter 4: The Test Engineering Manager     187
The Life of a TEM     187
Getting Projects and People     189
Impact     191
An Interview with Gmail TEM Ankit Mehta     193
An Interview with Android TEM Hung Dang     198
An Interview with Chrome TEM Joel Hynoski     202
The Test Engineering Director     206
An Interview with Search and Geo Test Director
Shelton Mar     207
An Interview with Engineering Tools Director
Ashish Kumar     211
An Interview with Google India Test Director Sujay Sahni     214
An Interview with Engineering Manager Brad Green     219
An Interview with James Whittaker     222

Chapter 5: Improving How Google Tests Software     229
Fatal Flaws in Google’s Process     229
The Future of the SET     231
The Future of the TE     233
The Future of the Test Director and Manager     234
The Future of Test Infrastructure     234
In Conclusion     235

Appendix A: Chrome OS Test Plan     237
Overview of Themes     237
Risk Analysis     238
Per-Build Baseline Testing     239
Per-LKG Day Testing     239
Per-Release Testing     239
Manual Versus Automation     240
Dev Versus Test Quality Focus     240
Release Channels     240
User Input     241
Test Case Repositories     241
Test Dashboarding     241
Virtualization     241
Performance     242
Stress, Long-Running, and Stability     242
Test Execution Framework (Autotest)     242
OEMs     242
Hardware Lab     242
E2E Farm Automation     243
Testing the Browser AppManager     243
Browser Testability     243
Hardware     244
Timeline     244
Primary Test Drivers     246
Relevant Documents     246

Appendix B: Test Tours for Chrome     247
The Shopping Tour     247
The Student Tour     248
Suggested Areas to Test     248
The International Calling Tour     249
Suggested Areas to Test     249
The Landmark Tour     249
Suggested Landmarks in Chrome     249
The All Nighter Tour     250
Suggested Areas to Test     250
The Artisan’s Tour     251
Tools in Chrome     251
The Bad Neighborhood Tour     251
Bad Neighborhoods in Chrome OS     251
The Personalization Tour     252
Ways to Customize Chrome     252

Appendix C: Blog Posts on Tools and Code     253
Take a BITE out of Bugs and Redundant Labor     253
Unleash the QualityBots     255
RPF: Google’s Record Playback Framework     257
Google Test Analytics—Now in Open Source     260
Comprehensive     260
Quick     260
Actionable     260
Sustained Value     260

Index     265