Web crawlers refer to a class of software that downloads pages, identifies the hyperlinks, and adds them to a database for future crawling. Crawlers are sometimes called web spiders, robots, worms, or wanderers and can be thought of as an automated text browser. A crawler’s downloaded pages are consumed by a scraper, which parses out certain pieces of information from those pages like hyperlinks to other pages.

A crawler can be written to be autonomous so that it populates its own list of fresh URLs to crawl, but is normally distributed across many machines and controlled centrally. Sample PHP crawler code is shown in Listing 18.1. These crawlers (which can be written in any language that is able to connect to the WWW) begin their work by having a list of URLs that need to be retrieved called the seeds. For a brand-new search engine, the initial seeds might be the URLs of web directories. Unlike an HTTP request from within a browser, the images, styles, and JavaScript files are not downloaded right away when a crawler downloads a page. The links to them, however, can be identified so that we can download those resources later.

class Crawler {
   private $URLList;
   private $nextIndex;
   function    construct(){
      $this->nextIndex=0;
      $this->URLList = array("http://SEEDWEBSITE/");
   }
   private function getNextURLToCrawl(){
      return $this->URLList[$this->nextIndex++];
   }
   private function printSummary(){
      echo count($this->URLList)." links. Index:".
         $this->nextIndex."<br>";
      foreach($this->URLList as $link){
         echo $link."<br>";
      }
   }
   // THIS CAN BE CALLED FROM LOOP OR CRON
   public function doIteration(){
      $url = $self->getNextURLToCrawl();
      // Do note crawl if not allowed
      if (robotsDisallow($url))
           return;
      echo "Crawling ".$url."<br>";
      //this function finds the <a> links
      scrapeHyperlinks($url);
      $self->printSummary();
   }
}

In the early days of web crawlers there was no protocol about how often to request pages, or which pages to include, so some crawlers requested entire sites at once, putting stress on the servers. Moreover, some sites crawled content that the author did not really want or expect to link on a public directory. These issues created a bad reputation for crawlers. As search engines began to take off, more and more crawlers appeared, indexing more and more pages.

To address the issue of politeness Martijn Koster, the creator of ALIWEB, drafted a set of guidelines enshrined as the Robots Exclusion Standard still used today.^2,3 These guidelines helped webmasters discourage certain pages from being crawled and indexed. The simple crawler in Listing 18.1 even adheres to it by calling the function robotsDisallow().

$DOM = new DOMDocument();
$DOM->loadHTML($HTMLDOCUMENT);

$aTags = $DOM->getElementsByTagName("a");
foreach($aTags as $link){
   echo $link->getAttribute("href")." - ".$link->nodeValue."<br>";
}

foreach($aTags as $link){
   $mailpos=strpos($link->getAttribute('href'),"mailto:");
   if($mailpos !== false){
      echo substr($link->getAttribute('href'),$mailpos+7)."<br>";
   }
}