Google and its competitors have created the first application to leverage the Database of Intentions in a commercial manner: paid search. Search tools are building the Database of Intentions. Search is also a catalyst in promising attempts at cracking one of mankind’s most intractable problems: the creation of artificial intelligence.
Development of search model as commercial practice is possible because of storage and bandwidth price reduction. Search straddles an increasingly complicated territory of marketing, media, technology, pop culture, international law and civil liberties. Danny Hillis from MacArthur Foundations believes that the future of search will be more about understanding, rather than simply finding.
Five questions about any topic before writing about it: who, what, where, why and when. And sixth is who’s making the money and how much?
How – search engine consists of three major pieces: the crawl, the index and the runtime system or query processor. The crawler is a specialized software program that hops from link to link on the WWW. It sends data back to a massive database called index. At the end of the day, the holy grail of all search engines is to decipher your true intent.
Who – almost everybody is searching the Web, but the younger you are and more education you have, the more you search.
What – 50% use two or three words, 20 % use just one. There is no greater act of creativity than the formation of a good question.
Where, why – searching is merely a means to an end. Search is about recovery (what we know it exist) and discovery (what we think it does exist and we didn’t find it yet).
When – Gerard Salton (Harvard and Cornell) introduced SMART (Salton’s Magical Automation Retriever of Text) in late 1960s, probably the first digital search engine.
First Internet search engine was probably Archie, a pre-Web search application created in 1990 by a McGill University student named Alan Emtage. WebCrawler from Brian Pinkerton was important to the evolution of search because it was the first to index the full text of the Web documents it found. AltaVista was the Google of its era. It was Louis Monier who took AltaVista from concept to executable code.
In 1995 number of companies working on search increased, but in 1996 it was still impossible to create a pure play in search that was economically viable. AltaVista was part of transaction of Compaq acquiring DEC. Companies like Lycos and Excite were introduced. Lycos introduced Web page summaries in search results. Vinod Khosla tried to persuade Excite founders to buy Google when it was still a research project. Excite introduced personalization with MyExcite. All the portals suffered from the classic business mistake of veering from their core mission. Unbeknownst to the all, there was a giant vacuum left in search. That vacuum would soon be filled by Google.
Jerry Yang and David Filo prepared Guide to the World Wide Web and the first iteration would later become Yahoo. That Guide made its debut in late 1994. Yahoo (Yet Another Hierarchical Officious Oracle). They build their company on exploration and discovery journey and not so much on intent based search. Tim Koogle CEO of Yahoo believed that search as a stand-alone service was very capital intensive – so much storage and bandwidth. The economics had not yet emerged to justify the investment.
Google was basically built on graph approach. Each computer was a node and each link on a Web page was a connection between nodes. Page told author that he didn’t have intention to create a search engine, but in order to create BackRub, they had to crawl the web. Tim Berners-Lee created a system to address need for connection and simplicity of access to information via network technology and hypertext and he created WWW. Page and Brin’s attempts to improve his WWW led to Google. The needle that threads these efforts together is citation – the practice of pointing to other people’s work in order to build up your own.
Page and Brin’s breakthrough was to create an algorithm – dubbed PageRank after Page – that manages to take into account both the number of links into a particular site and the number of links into each of the linking sites. The signal – now better known as PageRank – became the foundation of Google’s vaunted secret sauce.
The first attempt to license Google’s technology occurred very early in the project’s life. Vinod Khosla learned of Google through his connections, he tried to persuade Excite to make an offer. But Page wanted 1.6 million USD, Kosla was willing to push for 750.000 USD. Deal didn’t happen. They went to Andy Bechtolsheim, a founder of Sun, that write them a check for 100.000 USD and they needed to incorporate. But they moved forward, looking for a money and in the end, they persuade two most competitive top-tier firms – Sequoia Capital and Kleiner Perkins Caufiled & Byers to take the deal together. With VC money coming in, finance guys pushed for new CEO.
Another question was how to set up business model to start earning. Brin and Page were deeply suspicious of blending advertising and search. In 1999 they were burning money fast and they found a business model in another company, called GoTo.com. Bill Gross, founder saw in search the seeds of an economic revolution. His model was a billion dollars, one nickel at a time.
Company that he latter founded was Overture, company latter sold to Yahoo for 1.6 Billion USD. During his career he played with natural language interface, indexing (he created Magellan – early version of file manager), how people learn. But he eventually came to a challenge how to create a company that would compress time and leveraged people, a company that let businesses be conceived, prototyped and launched quickly. In 1996 IdeaLab was born. Capital markets stopped funding concept plays, and by the middle of 2001, investors wanted to wind down IdeaLab. A gem that was still in portfolio was Overture.
As the portals consolidated their grip on Internet traffic, demand from that traffic from independent e-commerce players soared. Acquiring traffic became expensive. But e-commerce players were paying for that traffic, even though they didn’t know if that traffic has any interest in them or not. Before Google, most engines employed simple keyword-based algorithms to determine ranking. Innovation in search languished and the tragedy of the commons prevailed: spammers quickly took control of the indexes – search engine spam.
Gross saw an opportunity here, to associate economic price with listings and by doing that, the system would start behave normally. He wanted to solve a problem of good traffic and crap. He realized it was a quality, not quantity that counted. He believed that to kill spam, one must add a friction of money into equation. He realized that maybe he can build an arbitrage system. He needed both the audience and advertisers – and the more advertisers the better. He was getting audience from Inktomi and he needed advertisers. He adopted the time-honored approach of dumping (first ones were almost for free). He introduced two innovative approaches for advertisers: performance-based model (advertisers paid for visitor only when a visitor clicked through an ad and onto the advertisers’ sites).
In the beginning he offered price of click, ten times lower than his costs, but he bet on market forces moving into directions of lifting prices over his costs. In auctions for certain keywords, price really did pick up. GoTo developed two lines of business, one on their own site and second providing search services to other companies for fee or split revenues (syndication business). In 2001, GoTo changed its name to Overture. But Overture decided to phase out their own site business not to conflict it with partners and that was a mistake. And Google was gaining ground – as a pure search destination. Google soon introduced AdWords and when they introduced pay-per-click model Overture sue them. Google also took AOL deal from Overture.
To profit from search and control its own destiny, a company requires three elements:
- High quality organic search results (algorithmic or editorial search).
- A paid search network.
- Own its own traffic – the consumer’s search queries against which editorial and paid results are displayed.
Google had all that. Microsoft, Yahoo and Overture was lacking certain elements. Yahoo and Microsoft started with acquisitions and their own development to tackle that ground.
At the end of 1999, Google spend per month was 0,5 Million and they had less than 20 Million in the bank. Company needed a business model that worked. They wanted to sell text-only ads to sponsors targeting particular keywords.
The crash of the banner advertising market and the meager revenues from Google’s first attempt at text advertising led Brin and Page to turn their gaze toward GoTo.com. In October 2000, Google introduced its new service, which it called AdWords. Initial versions of AdWords maintained the CPM approach advertisers still paid for impressions instead of clickthrough. In June 2000, Google replaced Inktomi as Yahoo’s core search service.
Google approach to marketing was to invest in the product and use PR as a tool for getting people to read and talk about Google. Google worked. Their approach, known as distributed computing, would soon become all the rage in corporate environment. PageRank patent is in fact owned by Stanford University, but licensed exclusively to Google until 2011.
Page and Brin could no longer fight off, pressure from investors to appoint outside CEO. So, they choose Eric Schmidt. But he shared his power with founders in a strange triumvirate.
Company was growing really fast and it was a question how might a company ensure that its original DNA remained intact. The vision Don’t Be Evil was created. It was a strange approach that created mixed feelings in business community. Google’s mission was: to organize the world’s information and make it universally accessible and useful.
Google growth was even outside core search projects, they had a list of 100 top projects, everybody could be working on. Small teams of engineers tackled hundreds of projects, all at once. In 2002, they improved AdWords with auctions and pay-per-click features.
By 2003, Google had more than 100.000 advertisers working with it, but still didn’t put too much into customer services. They preferred to automate their interactions with customers.
Google next introduced AdSense to allow other third-party publishers access to Google massive network of advertisers. But this concept is more based on content of a site than on intent-based queries.
In November 2003 Google tweaked its search result algorithms and it pushed a lot of organic clicks champions from first positions. This time SEO industry seemed to be a target. SEO seemed to flourish in Eastern Europe. Early SEO industry was about white and black hats. SEO firms were increasingly tempted to push the limits of what might be considered white-hat practices. Florida update of Google search showed that Google decided that affiliate and SEO spam had reached unacceptable levels. It was now time when engines like Google decide what is relevant. That at the end increased revenue for AdWords like programs.
Paid search shifted the marketing model from one based on content attachment to one based on intent attachment. Magic of intent-based marketing is that it shifts marketing dollars from the unknown to the knowable. Search turns a cost center into a profit center. In a post-Web world, the model for news is no longer site driven. Publisher that do not offer additional paid subscriptions benefits beyond the articles themselves are not paying attention to the needs of their communities.
Because of its innovative and relatively new business model, search is testing the boundaries of how business works in several ways. Cases from American Blinds and Playboy (against Netflix) showed that there was a potential issue with companies having brand protection, but not being protected from keywords business model. Google pushed their policy for better search by declaring that they will sell any trademark term. Trademark terms are the verbs of commercial speech to them.
Another issue with Google business model could be click-fraud activities, that can jeopardize their AdSense revenue streams.
When Google moved into other areas from search, areas like e-mail and productivity suits, question was again privacy. E-mail was used as another record in Database of Intentions. But privacy issues have grown also in other domains. The Patriot Act in America allowed government to tap a suspect’s clickstream. This act also loosened standards for who is a suspect and how government informs a suspect that he is being under investigation.
Google was also under pressure, like other foreign companies, trying to enter Chinese market. Everybody wants to be there because of market size, but China is allowing businesses to run, under their own rules. Google was quite cooperative with them. Having share in Baidu and wanting to be present of that market, they cooperate with government demands.
At one-point Google needed to move to IPO. Not because they really wanted, but because under SEC rules, they were forced to, since they have given stock options to more than one thousand of its employees. But they designed a corporate structure, where investors had little control over business, creating dual class shareholding structure, with founders and senior executives holding control over common shareholders. Another mistake in IPO process was an interview in Playboy from Brin and Page one week prior to IPO. They went public in August 2004, with price lower than they expected, but then stock started to grow.
After the IPO Google started to work on their after-IPO strategy review. They organized more according to products and functions, but still two monarch Larry and Sergey were running the show.
The biggest difference between Google and Yahoo is that Google sees solving information business with technology, Yahoo is seeing humans as integral to the solution. Google is saying, we are media-driven technology company. Both Yahoo and Google are converging into the same space; they mediate information and services for consumers and derive value from those services using the traditional revenue stream of the media business – advertising and subscriptions.
In a perfect market, where demand is simply one computable bit of information and supply another, matching the two is an extremely lucrative business. Google is angling to become the de facto marketplace for all of global commerce, unseating eBay in the process.
The search engine of the future isn’t really a search engine as we know it, it is more like an intelligent agent. Eventually everything of value will be connected to the Web, because to be connected is definitional to the concept of value in a wired world. Ubiquity is critical to perfect search, but it means nothing if the engine does not understand you. Clickstreams are the seeds that will grow into our culture’s own memex – a new ecology of potential knowledge – and search will be the spade that turns the Internet’s soil.
A9 from Amazon was the first engine to employ the concept of search history in its results. A9 broken search into recovery and discovery. Many in search industry believe that search will be revolutionized by what is called metadata. Semantic Web is a concept that is not new, but it can be potential of Web development. IBM tried to work on that development with a project called WebFountain. A lot of hardware and a lot of metadata tagging. It basically structures the Web. Domain-based searches are another way forward.
Perfect search is about:
- Ubiquity (integration of information into Web indexes)
- Personalized search (using data about you to improve relevance of your search)
- The rise of semantic Web (tagging of information)
- Domain-specific search
- Web time axis
There are two main conflicts that are growing in search business and in almost every web-based business model. First is conflict of intellectual property, conflict of owning the rights to distribute content. Another issue is who pays the freight for all the information that is driven across the Internet. The last one has to do with net neutrality (non-discrimination, interconnection and access).