Google's Innovation Pathway: Some Classical Ingredients

Everyone acknowledges the role of "serendipity" in the progress of science and technology, where the term means "making desirable discoveries by accident."  Picture it as Google co-founder Sergy Brin enjoying zero-gravity at left - a major breakthrough is often something you bump into while happily floating somewhere else.

Google has marched from a search algorithim developed by graduate students at Stanford to a search engine and into a dominant position in web advertizing and onward to a global information empire now encompassing the past as well as the future -- with a diplomacy wing being incorporated into Google Ideas.  In spite of Google's ubiquity in the global information cloud, was its success in fact an accident? 
The question arises in Charles Petersen's in the New York Review of Books of business writer Ken Auletta's history of Google.  Brin and his graduate program friend Larry Page began to work with Page's desire to understand every page of every site as the target of a set of links that arrived there from anywhere else in the web.  They developed a method of mapping these backlinks which they called Backrub, and in a story that has been well-told before, particularly by Wired's John Batelle, they developed the PageRank algorithm that created a search engine that was better than any then available.

Page really did grasp something that has become a fairly common idea since: you will get less noise in your result if you supplement a search for a word or a word string with an evaluation of the other sites that have linked to it.  He saw the web as a structure of citations, and mapping the citations introduced the judgements of hundreds, thousands, or millions of other people about the value of that site's use of the word or word string. Terms like "the wisdom of crowds" or "crowdsourcing" capture the idea of collective intelligence that Brin and Page used as the central principle of their search algorithm.

Except in fact there was no search algorithm when Page was thinking about graphing the web. In the 2009 Commencement speech he gave at his alma mater the University of Michigan, he said,

I had one of those [vivid] dreams when I was 23. When I suddenly woke up, I was thinking: what if we could download the whole web, and just keep the links and... I grabbed a pen and started writing! Sometimes it is important to wake up and stop dreaming. I spent the middle of that night scribbling out the details and convincing myself it would work. Soon after, I told my advisor, Terry Winograd, it would take a couple of weeks to download the web -- he nodded knowingly, fully aware it would take much longer but wise enough to not tell me. The optimism of youth is often underrated! Amazingly, I had no thought of building a search engine. The idea wasn't even on the radar. But, much later we happened upon a better way of ranking webpages to make a really great search engine, and Google was born. When a really great dream shows up, grab it!

Page is identifying some classic ingredients of creative breakthroughs:

• disconnection from specific goals: the "dream" is pure curiosity, unharnessed by any aspect of reality.
• absence of a sense of useful application
• discontinuity with the "mature" idea

Page's advisor Terry Winograd gave Brin and Page lots of rope to run with, Stanford gave them enormous bandwidth, and they came up with various tricks and wheedlings to obtain lots of cheap equipment. Thus a further classical element is

• quasi-free resources - no strict cost-accounting

 These are features that are more easily sustained in academic than in commercial environments. Highly creative commercial environments like the Eastman Kodak of Kenneth Mees, wartime Lockheed, monopoly-era Bell Labs, and perhaps like Google today, set up university-style environments with free space for playing around, daydreaming, and working obsessively on compelling ideas that look entirely marginal to the main business. 

A later conceptual leap connected the mathematics of worldwide links to web searches. And then a further major breakthrough consisted of Brin and Page's response to their attempt to sell their algorithm to Yahoo! so they could go back to the important work of finishing their PhDs.  Yahoo! didn't want their search engine, for a reason Auletta explains:

[The Yahoo founders] were impressed with [Google’s] search engine. Very impressed, actually; their concern was that it was too good…. The more relevant the results of a search were, the fewer [pages] users would experience before leaving Yahoo. Instead of ten pages, they might see just a couple, and that would deflate the number of page views Yahoo sold advertisers.

Brin and Page are sent away, and the crucial next step is their reaction to this defeat: they "disdained" the reasoning behind it (Petersen), and continued on their own path.  That is to say that they disdained the core business model of the commercial Internet of the time: rather than following what the market seemed to say, they rejected it.

It is easy in retrospect to see this as an act of enormous courage or genius. Let's posit that it was both, and then define what genius and courage meant in such a context: a kind of perversity and youthful defiance ("screw you, Yahoo!"), and the confidence or simple stubbornness or persisting in spite of a lack of alternatives against the industry's conventional wisdom.   Brin and page had in fact to reject the direct relevance of the business model as such to the development of an important idea: they "decided to continuing improving Google's search algorithms," on the tacit assumption that they would not be helped by following the market at least in the short run.

Petersen picks up the next phase:

It wasn’t until Google grew desperate for funding during the dot-com bust of the early 2000s that Page, Brin, and their colleagues began to see that the technical advantage they had gained over other search engines might translate into an economic advantage as well. Aside from page views, one of the few easily measured statistics on the early Web was “click-throughs,” the number of times visitors to a site found an ad displayed enticing enough to click on it, and then be taken to the advertiser’s own website, where the product or service in question might be purchased or used. Most websites, including those of other search engines, found that they could earn more by charging small amounts for each time an ad was seen (page views) rather than charging a larger amount for the far less frequent instances when a visitor clicked on an ad (click-throughs) and then visited the advertiser’s own website.

 Google changed this. A conventional interpretation might be that financial necessity is the mother of invention. But Brin and Page had been desperate for money at most phases of the development, so financial necessity was nothing new.  Instead, it's worth emphasizing again that some combination of their experience and their personal outlooks allowed them to accept that market potential is over the horizon w hen development was most important.  This meant that in spite of the pressure at the time, they didn't take short cuts. This allowed a more recessive but important element of the innovation process to emerge. For the innovation to have its most important impact, the innovation has to change the business model that at the time controls the market.  This is finally what "click-throughs" did.

None of this is to whitewash either the innovation process as it must be practiced under Silicon Valley rules, or to give a free pass to Google's current practices and impact.  Petersen is especially concerned about privacy issues, and I am especially concerned about the free circulation of information under Google globalization.  By making Internet advertizing massively more effective, Google made the commercialization of the Internet more pervasive, far more so than during the Netscape Era in which the founders started their doctorates.  But even here Google offers a counterintuitive lesson, and it is built into the advertizing model itself.

With Google’s search engine advertising, the value of the ads comes less from their quantity (the number of times the ads appear, for which the company receives nothing) than from their quality (the number of times the ads match what users are looking for and attract purposeful clicks, for which Google receives fees).

Averse as I am to sponsored links, Petersen's description here is right.  There's an open source idealism lurking in the business model Google finally hit on, one that makes unfake the often repeated assertion of the Google elders that "Our mission is to organize the world's information and make it universally accessible and useful." The idealism doesn't fit into its corporate embodiment, and yet the idealism is what make the company so successful, and made it's innovation possible.

A summary of the innovation elements would run like this:

1. the dream, disconnected from specific goals; pure curiosity
2. long absence of a sense of useful application; discontinuity with the "mature" idea
3. quasi-free resources - no strict cost-accounting during long "precommercial" phase
4. frame jumping (from graphs to web searches)
5. rejecting the dominant business model
6. changing the business model to fit the innovation, not fitting the innovation to the dominant model
7. pursuit of maximum social value, not just (internalizable) market value

All of these come with enough gaps to deter just about anyone - anyone except people that have overpowering interests untethered from markets of the time.

Basic Research at . . . Microsoft!

The New York Times has a piece announcing that Microsoft will open its sixth "basic research" lab in July, this one headed by the mathematician Jennifer Tour Chayes (at left) and placed next door to MIT. Dr. Chayes will apparently "be one of the first women to direct a research lab run by an American corporation."

In the last couple of decades, industy has tended to close or spin-off its basic research labs, to rely on partnerships with academia, and to conduct what Intel calls "directed research" with two important features: 1) it is oriented toward short-term product development and 2) it maximizes its use of other people's research rather than developing everything in-house. Henry Chesbrough has dubbed this system "open innovation," and contrasts it with the in-house lab model. Is Microsoft partially reversing this trend?

The quick answer is no. Microsoft Research has 800 employees with doctorates. Google has 100. In its heyday, Bell Labs had well over a dozen major facilities, the largest of which, at Naperville-Lisle outside Chicago, had 11,000 employees.

Size isn't the only thing that matters. But we do know enough about the innovation process to know that randomness is a central variable - people get amazing data or have great insights at unexpected and unpredictable moments. Size does matter. Not in a linear way, where big = best, but in an unpredictable and yet semi-regular way. Throw in the fact that MSFT and GOOG are two of the wealthiest companies in history, and you are likely looking at an exception to the industry trend of oursourcing research rather than a trend that counters it.

And yet Microsoft Research is a tribute to the economic as well as the strategic value of basic research. As public university budgets are continuously squeezed, universities can afford less basic research. Given the country and the world's desperate research needs, this would be a good time for large companies - even those riding the revenue roller-coaster like Intel - to fund more basic than they are funding now.

P.S. I've avoided blogging on the Microsoft bid for Yahoo! because it makes me, well, sad. I started graduate school the same year that the IBM PC came out - 1981. As an academic I grew up with DOS, and snubbed Apple's graphical interface as kid stuff for almost 15 years. Now Microsoft and Windows are dated and stagnant, and Microsoft's amazing cash reserves (perhaps still as high as $30 billion before this offer) are a triumph of monopoly power over innovation. Joe Nocera at the New York Times sums all this up quite nicely.