Parsing web Pages with php

For this not-so-easy job we’ll use PHP Simple HTML DOM Parser.

The latest version is available at sourceforge.net.

As of today the latest is version 1.5. That’s the one we’ll download:

wget -O simplehtmldom_1_5.zip http://downloads.sourceforge.net/project/simplehtmldom/simplehtmldom/1.5/simplehtmldom_1_5.zip?r=http%3A%2F%2Fsourceforge.net%2Fprojects%2Fsimplehtmldom%2Ffiles%2Fsimplehtmldom%2F1.5%2F\&ts=1433252429\&use_mirror=softlayer-ams

Downloaded, now unpack it into a folder accessible over the web, say /var/www/html/webparser:

mkdir /var/www/html/webparser  
mv simplehtmldom_1_5.zip /var/www/html/webparser/  
cd /var/www/html/webparser  
unzip simplehtmldom_1_5.zip

Based on the example from the parser’s page, calling a file with the following content will output all img objects on the homepage of mysite.com

<?php
include('simple_html_dom.php');
$html = file_get_html('http://www.mysite.com/');

foreach($html->find('img') as $element)
  echo $element->src . '<br>';
?>

Calling a file with the following content will output all links from the homepage of mysite.com

<?php
include('simple_html_dom.php');

$html = file_get_html('http://www.mysite.com/');

foreach($html->find('a') as $element)
   echo $element->href . '<br>';
?>

Don’t even try var_dump($element), since it’ll hang your computer and/or the server you deployed the parser on.

Armed with test data, I decided to parse the habrahabr sandbox. The following code outputs the first ten posts:

<?php
include('simple_html_dom.php');

$html = file_get_html('http://habrahabr.ru/sandbox/');

$cnt = 10;
foreach($html->find('.post') as $post) {
  foreach($post->find('h1') as $header) {
    echo str_replace('a href="','a href="http://habrahabr.ru',$header->find('a')[0]) . '<br>';
    $cnt--;
  }
  if ($cnt <= 0){
    break;
  }
}
?>

Handling links with an h2 tag, using tecmint.com as an example:

<?php
include('simple_html_dom.php');

$html = file_get_html('http://www.tecmint.com/');

foreach($html->find('h2') as $post)
  echo $post->find('a')[0] . '<br>';
?>