The endless testing

Google’s Search algorithm is one of the most fascinating analytical initiatives (or should I call it a project) that exist in our field today. Enough books and articles have been written about it that linger between unveiling its inner workings and increasing the cloud of its secrecy. The weekly meetings of all of Google’s critical Search Engineers in the never ending attempt of improving the quality of its search results are probably one of the most interesting meetings, close in importance and impact on people’s daily lives to the president’s weekly “Mood of the nation” briefings and discussions.

One of the more intriguing elements of this weekly meeting and for me the foundation of Google’s long lasting superiority resides in the changed philosophy of how to do testing. Its core principle is that every search query is part of at least one test (more likely multiple) there is no separation anymore between non-tested activities and the usual small percentage of separated space of testing. Wired editor Steven Levy describes it well in this month’s WIRED:

“”There are so many changes to measure that Google has discarded the traditional scientific nostrum that only one experiment should be conducted at a time. ‘On most Google queries, you are actually in multiple control or experimental groups simultaneously’ says search quality engineer Patrick Riley. Then he corrects himself. ‘Essentially,” he says, ‘all the queries are involved in some test’. In other words, just about every time you search on Google, you are a lab rat.”

It’s a fascinating detour from most marketers more conservative testing philosophy where testing only happens in a well shielded space. I am curious to see how we could translate this Google approach of 100% covered, constant, and multiple testing to other marketing activities.