Grab the gist with the complete, working source code. Benchmark it against the one featured on the previous article:

$ \time perl mojo-crawler.pl
23.08user 0.07system 2:11.99elapsed 17%CPU (0avgtext+0avgdata 84064maxresident)k
16inputs+0outputs (0major+6356minor)pagefaults 0swaps

$ \time perl yada-crawler.pl
8.83user 0.30system 0:12.60elapsed 72%CPU (0avgtext+0avgdata 131008maxresident)k
0inputs+0outputs (0major+8607minor)pagefaults 0swaps

How can it be 10x faster while consuming less than a half of CPU resources?!

Sorry, I had cheated a bit on the mojo-crawler.pl benchmark results. It implicitly uses the EV, a high performance full-featured event loop library when it is present. And it is not required for Mojolicious to work properly. Let's disable it:

$ MOJO_REACTOR=Mojo::Reactor::Poll time perl mojo-crawler.pl
113.99user 13.37system 2:08.46elapsed 99%CPU (0avgtext+0avgdata 83808maxresident)k
2912inputs+0outputs (18major+5789minor)pagefaults 0swaps

The elapsed time is the same with ou without EV, but now the pure-Perl crawler hogs the CPU!

Why? EV provides an interface to libev, which clearly does a better connection polling job than the 100% interpreted code. The bridge between Perl and the compiled library is called XS:

XS is an interface description file format used to create an extension interface between Perl and C code (or a C library) which one wishes to use with Perl.

Actually, CPAN is full of high performance XS-based modules for many tasks:

• Net::Curl - Perl interface for libcurl;
• HTTP::Parser::XS - a fast, primitive HTTP request parser;
• JSON::XS - JSON serialising/deserialising, done correctly and fast;
• Unicode::CheckUTF8 - an XS wrapper around some Unicode Consortium code to check if a string is valid UTF-8;
• XML::LibXML - Perl Binding for libxml2.

Thus, an efficient and fast web crawler/scraper could be constructed with those "bare-metal" building blocks ;)

Show me the code!

#!/usr/bin/env perl
use 5.016;
use common::sense;
use utf8::all;

# Use fast binary libraries
use EV;
use Web::Scraper::LibXML;
use YADA 0.039;

YADA->new(
    common_opts => {
        # Available opts @ http://curl.haxx.se/libcurl/c/curl_easy_setopt.html
        encoding        => '',
        followlocation  => 1,
        maxredirs       => 5,
    }, http_response => 1, max => 4,
)->append([qw[
    http://sysd.org/page/1/
    http://sysd.org/page/2/
    http://sysd.org/page/3/
]] => sub {
    my ($self) = @_;
    return  if $self->has_error
        or not $self->response->is_success
        or not $self->response->content_is_html;

    # Declare the scraper once and then reuse it
    state $scraper = scraper {
        process q(html title), title => q(text);
        process q(a), q(links[]) => q(@href);
    };

    # Employ amazing Perl (en|de)coding powers to handle HTML charsets
    my $doc = $scraper->scrape(
        $self->response->decoded_content,
        $self->final_url,
    );

    printf qq(%-64s %s\n), $self->final_url, $doc->{title};

    # Enqueue links from the parsed page
    $self->queue->prepend([
        grep {
            $_->can(q(host)) and $_->scheme =~ m{^https?$}x
            and $_->host eq $self->initial_url->host
            and (grep { length } $_->path_segments) <= 3
        } @{$doc->{links} // []}
    ] => __SUB__);
})->wait;

Now what?!

The example above has half the lines of code of the previous one. This comes at a cost of installing a bunch of external dependencies from the CPAN:

$ cpanm AnyEvent::Net::Curl::Queued EV HTML::TreeBuilder::LibXML Web::Scraper utf8::all

Despite the use 5.016 pragma, this code works fine on Perl 5.10 if you get rid of the __SUB__ reference.

So, what approach is the better one? Obviously, it depends. There is no silver bullet: web crawling is ultimately I/O-bound! However, specialized and well-tested libraries guarantee the I/O-boundness. For instance, trimming the ::LibXML part from the use Web::Scraper::LibXML statement considerably slows down our tiny crawler, because the HTML parsing will allocate more CPU cycles than the connection polling.

As the edge case, let's see how the venerable GNU Wget tool (see also yada, which comes bundled together with the AE::N::C::Queued distribution) behaves:

$ "time" wget -r --follow-tags a http://sysd.org/
0.23user 0.41system 1:10.20elapsed 0%CPU (0avgtext+0avgdata 23920maxresident)k
0inputs+40704outputs (0major+4323minor)pagefaults 0swaps

Despite it's clear disadvantage of using single connection, it is almost completely I/O-bound since it's URL extraction code doesn't require complete parsing of the HTML.

Google Refine is awesome. If you're unaware of what it is, access their official page and watch at least the first screencast. You'll see it can be helpful for several ETL-related tasks.

Currently, I use it a lot, specially for simple (but boring) tasks, like loading a CSV, trimming out some outliers and saving as JSON to be imported into MongoDB. Nothing a Perl one-liner couldn't do.

However, the opposite is not true: Perl one-liners are a lot more flexible than Google Refine. Now, what if we could merge both?

As a practical example, I'll use some georeferenced data I was working at. Let's suppose I have to deduplicate registers, and one of "duplicate" rules is their proximity on the map. Google Refine is far from a full-featured GIS, and is unable to handle bidimensional coordinate system. Enter the GeoDNA: an algorithm to lower geospatial dimensions. As it's FAQ says,

GeoDNA is a way to represent a latitude/longitude coordinate pair as a string. That sounds simple enough, but it's a special string format: the longer it is, the more accurate it is. More importantly, each string uniquely defines a region of the earth's surface, so in general, GeoDNA codes with similar prefixes are located near each other. This can be used to perform proximity searching using only string comparisons (like the SQL "LIKE" operator).

Another interesting property of GeoDNA is that when ordening a set of records by their GeoDNA code, close locations are likely to appear in adjacent rows (sometimes, close locations will share very different prefixes, but similar prefixes always represent close locations).

To incorporate GeoDNA into Google Refine, we'll use the Add column by fetching URLs option, clicking on the header of any column (which column it will be doesn't matter as we'll use two of them, anyway):

As the expression, we'll paste the following code (here, pay attention to the correct latitude/longitude column names):

'http://127.0.0.1:3000/?lat='+
row.cells['latitude'].value
+'&lon='+
row.cells['longitude'].value

Throttle delay can be zeroed, as our webservice is local. The final configuration should look like this (don't push the OK button, yet):

Now, check if you have Mojolicious and Geo::DNA Perl modules (install them via CPAN, if not) and paste into your terminal:

perl -MGeo::DNA -Mojo -E 'a("/"=>sub{my$s=shift;$s->render(json=>{geocode=>Geo::DNA::encode_geo_dna($s->param("lat"),$s->param("lon"))})})->start' daemon

If you prefer a "human-readable" version, paste the following code into geocode-webservice.pl:

#!/usr/bin/env perl
use Geo::DNA qw(encode_geo_dna);
use Mojolicious::Lite;
any '/' => sub {
    my $self = shift;
    $self->render(json => {
        geocode => encode_geo_dna(
            $self->param('lat'),
            $self->param('lon'),
        ),
    });
};
app->start;

Once you started a webservice, it will report Server available at http://127.0.0.1:3000. Now, click OK on Google Refine dialog and wait. Even without delay, it could be a bit slow; however, even then this hack saved me a lot of time ;)