Does Whatever a Spider Can and Hitting the Links

A search engine spider does the search engine's grunt work: It scans Web pages and creates indexes of keywords. Once a spider has visited, scanned and categorized a page, it follows links from that page to other sites. The spider will continue to crawl from one site to the next, which means the search engine's index becomes more comprehensive and robust. To learn more about these programs, read How Search Engines Work.

Google uses lots of tricks to prevent people from cheating the system to get higher placement on SERPs. For example, as a Web page adds links to more sites, its voting power decreases. A Web page that has a high PageRank with lots of outgoing links can have less influence than a lower-ranked page with only one or two outgoing links.

The Google Search Engine

­Google's search engine is a powerful tool. Without search engines like Google, it would be practically impossible to find the information you need when you browse the Web. Like all search engines, Google uses a special algorithm to generate search results. While Google shares general facts about its algorithm, the specifics are a company secret. This helps Google remain competitive with other search engines on the Web and reduces the chance of someone finding out how to abuse the system.

­Google uses automated programs called spiders or crawlers, just like most search engines. Also like other search engines, Google has a large index of keywords and where those words can be found. What sets Google apart is how it ranks search results, which in turn determines the order Google displays results on its search engine results page (SERP). Google uses a trademarked algorithm called PageRank, which assigns each Web page a relevancy score.

A Web page's PageRank depends on a few factors:

  • The frequency and location of keywords within the Web page: If the keyword only appears once within the body of a page, it will receive a low score for that keyword.
  • How long the Web page has existed: People create new Web pages every day, and not all of them stick around for long. Google places more value on pages with an established history.
  • The number of other Web pages that link to the page in question: Google looks at how many Web pages link to a particular site to determine its relevance.

Out of these three factors, the third is the most important. It's easier to understand it with an example. Let's look at a search for the terms "Planet Earth."

As more Web pages link to Discovery's Planet Earth page, the Discovery page's rank increases. When Discovery's page ranks higher than other pages, it shows up at the top of the Google search results page.

Because Google looks at links to a Web page as a vote, it's not easy to cheat the system. The best way to make sure your Web page is high up on Google's search results is to provide great content so that people will link back to your page. The more links your page gets, the higher its PageRank score will be. If you attract the attention of sites with a high PageRank score, your score will grow faster.

Google initiated an experiment with its search engine in 2008. For the first time, Google is allowing a group of beta testers to change the ranking order of search results. In this experiment, beta testers can promote or demote search results and tailor their search experience so that it's more personally relevant. Google executives say there's no guarantee that the company will ever implement this feature into the search engine globally.­

Google offers many different kinds of services in addition to chat. In the next section, we'll see how some of them work.