Three Things I Learned about Testing at Google that Might Surprise You
Like this article? We recommend
Like this article? We recommend
Like this article? We recommend
When I first set out to write How Google Tests Software, I was a test engineering director over a number of Google products including Chrome, Maps, and Cloud. My goal was to document all the testing goodness that was going on inside the company, and although I accomplished that goal, what I really learned was far more cathartic. By the time I finished the book, Google had undergone a major transformation from a centralized testing organization to a distributed test model where testers reported to their development managers.
In retrospect, the centralized testing model described in the book was the right approach for a growing Google. The power of centralization meant that Test could take on a more visible role within the company, share resources and experiences among teams, and have enough critical mass to make quality something hard to ignore. It worked, and the book describes how this organization was built and how it executed on the massive scale that is Google. There are lessons here that every company from startups to medium and large organizations needs to hear.
But in the end, the test organization at Google grew too big and wielded too much power. Battles between product groups demanding more testers and the test team denying them based on the logic that too many testers were a crutch were legendary. “Test” became a subculture within the company and this brings me to my first lesson:
1. Whenever testers identify more with their discipline than with their product, it’s not healthy
I found an old t-shirt in a schwag closet at Google that sums up this attitude: I TEST THEREFORE I AM. This is categorically unhealthy. Teams at Google where testers celebrated their bugs were the least functional teams we had. Please destroy these shirts. Throw away the “top bug” awards and have your team -- the whole team -- get behind the product. You know you are successful when every PM, developer, tester and technical writer cites the product and not their role when asked, “What do you work on?”
2. Testing does not scale
Think about products like Google Maps. The only person capable of finding a bug in the location or address of his home is the person who lives there. No tester you can hire will find this problem. The best testers can only find a few really important bugs a week. To scale testing you have to do two things:
- reduce the size of the bug pool with intense unit and developer testing
- figure out how to do testing in production through instrumentation, beta testing or crowd sourcing
At Google we made everyone into a tester of our maps products by forcing new versions of Google Maps onto every internal employee. If you were on the corp net and used Google Maps, you got our test version automatically. Then we made it easy -- one-click easy -- to report bugs directly into our bug database. Suddenly we had tens of thousands of real users who were also effective testers. This beats tens of testers pretending to be users any day.
3. Testing is only part of the quality process
At Google, technologies like continuous builds, centralized test infrastructure, code reviews and so forth formed a holistic quality strategy that, when augmented with testing, put out pretty good software. If you rely too heavily on testing, you are very likely to come up short.
All three of these and many more aspects of Google’s testing process are in the book. Happy reading!