In keeping with this year's Dancer Advent Calendar trend, the example app will be built on Dancer 2, but it should work just as well with Dancer 1.

Alright, let's get to work.

Our web application will be used to search through documents stored in a MySQL database. We'll use a simple table with the following structure:

CREATE TABLE documents (
    id int NOT NULL AUTO_INCREMENT,
    title varchar(200) NOT NULL,
    contents_text text NOT NULL,
    contents_html text NOT NULL,
    PRIMARY KEY (id)
);

Each document has an unique ID, a title, and contents, stored as both plain text and as HTML. We need the two formats for different purposes -- HTML will be used to display the document in the browser, while plain text will be fed to the indexing mechanism of the search engine (because we do not want to index the HTML tags, obviously).

We can populate the database with any kind of document data -- for my test version, I used a simple script to fill the database with POD documentation extracted from Dancer distribution. The script is included at the end of this article, in case you'd like to use it yourself.

Installation and Configuration of Sphinx

Sphinx can be installed pretty easily, using one of the pre-compiled .rpm or .deb packages, or the source tarball. These are available at the download page at SphinxSearch.com -- grab the one that suits you and follow the installation instructions.

When Sphinx is installed, it needs to be configured before we can play with it. Its main configuration file is usually located at /etc/sphinx/sphinx.conf. For our purposes, a very basic setup will do -- we'll put the following in the sphinx.conf file:

source documents
{
    type        = mysql
    sql_host    = localhost
    sql_user    = user
    sql_pass    = hunter1
    sql_db      = docs

    sql_query   = \
        SELECT id, title, contents_text FROM documents
}

index test
{
    source          = documents
    charset_type    = utf-8
    path            = /usr/local/sphinx/data/test
}

This defines one source, which is what Sphinx uses to gather data, and one index, which will be created by processing the collected data and will then be queried when we perform the searches. In our case, the source is the documents database that we just created. The sql_query directive defines the SELECT query that Sphinx will use to pull the data, and it includes all the fields from the documents table, except contents_html -- like we said, HTML is not supposed to be indexed.

That's all that we need to start using Sphinx. After we make sure the searchd daemon is running, we can proceed with indexing the data. We call indexer with the name of the index:

$ indexer test

It should spit out some information about the indexing operation, and when it's done, we can do our first search:

$ search "plugin"

index 'test': query 'plugin ': returned 8 matches of 8 total in 0.002 sec

displaying matches:
1. document=19, weight=2713
2. document=44, weight=2694
3. document=20, weight=1713
4. document=2, weight=1672
5. document=1, weight=1640
6. document=13, weight=1640
7. document=27, weight=1601
8. document=28, weight=1601

Apparently, there are 8 documents in the Dancer documentation with the word plugin, and the one with the ID of 19 is the highest ranking result. Let's see which document that is:

mysql> SELECT title FROM documents WHERE id = 19;
+----------------------------------------------------+
| title                                              |
+----------------------------------------------------+
| Dancer::Plugin - helper for writing Dancer plugins |
+----------------------------------------------------+

It's the documentation for Dancer::Plugin, and it makes total sense that this is the first result for the word plugin. Sphinx setup is thus ready and we can get to the web application part of our little project.

The Basic Application

We'll start with a simple web application (let's call it DancerSearch) that just shows a search form, and then we'll extend it with more features. It will be using Dancer 2.0, and the Dancer::Plugin::Database plugin (we'll use it to access the documents database). The code below is the initial lib/DancerSearch.pm file:

package DancerSearch;

use Dancer 2.0;
use Dancer::Plugin::Database;

get '/' => sub {
    template 'index';
};

1;

We're also going to need a little startup script, bin/app.pl:

#!/usr/bin/env perl

use Dancer 2.0;
use DancerSearch;

start;

And a simple layout -- views/layouts/main.tt:

<!doctype html>
<html>
    <head>
    <title>Dancer Search Engine</title>
    <link rel="stylesheet" href="css/style.css">
    </head>
    <body>
    <h1>Dancer Search Engine</h1>
    <div id="content">
        [% content %]
    </div>
    </body>
</html>

And, of course, a template for our index page. For now it'll just contain a search form -- views/index.tt:

<form action="/" method="get">
    Search query:
    <input type="text" name="query">
    <input type="submit" value="Search">
</form>

Last but not least, we need a configuration file to tell our app which layout we want to use, and how to connect to our documents database using the Dancer::Plugin::Database plugin. This goes into config.yml:

layout: main

plugins:
    Database:
        driver:     mysql
        host:       localhost
        database:   docs
        username:   user
        password:   hunter1

We can now launch the application, and it will greet us with a search form. Which, unsurprisingly, doesn't work yet. Let's wire it up to Sphinx.

The Sphinx::Search CPAN Module

There is a CPAN module called Sphinx::Search that provides a Perl interface to Sphinx, and we're going to use it in our app. We put use Sphinx::Search in DancerSearch.pm, and add the following piece of code before the get '/' route handler:

# Create a new Sphinx::Search instance
my $sph = Sphinx::Search->new;
# Match all words, sort by relevance, return the first 10 results
$sph->SetMatchMode(SPH_MATCH_ALL);
$sph->SetSortMode(SPH_SORT_RELEVANCE);
$sph->SetLimits(0, 10);

This creates a new instance of Sphinx::Search (which will be used to talk to the Sphinx daemon and do the searches), and sets up a few basic options, such as how many results should be returned and in what order. Now comes the most interesting part -- actually performing a search in our application. We insert this chunk of code at the beginning of the get '/' route handler:

if (my $phrase = params('query')->{'phrase'}) {
    # Send the search query to Sphinx
    my $results = $sph->Query($phrase);

    my $retrieved_count = 0;
    my $total_count;
    my $documents = [];

    if ($total_count = $results->{'total_found'}) {
        $retrieved_count = @{$results->{'matches'}};
        # Get the array of document IDs
        my @document_ids = map { $_->{'doc'} } @{$results->{'matches'}};
        # Join the IDs to use in SQL query (the IDs come from Sphinx, so we
        # can trust them to be safe)
        my $ids_joined = join ',', @document_ids;

        # Select documents, in the same order as returned by Sphinx
        # (the contents of $ids_joined comes from Sphinx)
        my $sth = database->prepare('SELECT id, title FROM documents ' .
            "WHERE id IN ($ids_joined) ORDER BY FIELD(id, $ids_joined)");
        $sth->execute;

        # Fetch all results as an arrayref of hashrefs
        $documents = $sth->fetchall_arrayref({});
    }

    # Show search results page
    return template 'index', {
        phrase => encode_entities($phrase),
        retrieved_count => $retrieved_count,
        total_count => $total_count,
        documents => $documents
    };
}

Let's go through what is happening here. First, we check if there was actually a search phrase in the query string (params('query')->{'phrase'}). If there was one, we pass it to the $sph->Query() method, which queries Sphinx and returns the search results (the returned data structure is briefly explained in the description of the Query method in Sphinx::Search documentation).

We then check the number of results ($results->{'total_found'}), and if it's greater than zero, it means we found something and we need to retrieve the documents data from the database. Sphinx only returns the IDs of the matching documents (as shown earlier in the test search that we did using the command line), so we need to send a query to the database to get the actual data, such as document titles that we want to display in the results (note that we're using the ORDER BY FIELD construct in the SELECT query to maintain the same order as the list returned by Sphinx).

When we have the documents data ready, we pass it along with other information (such as the total number of results) to be displayed in our index template. But, hold on a second -- the template is not yet ready to display the results, it only shows the search form. Let's fix that now -- below the search form, we add the following code:

[% IF phrase %]
    <p>Search results for <strong>&quot;[% phrase %]&quot;</strong></p>

    [% IF total_count %]
    <p>
        Found [% total_count %] hits.
        Showing results 1 - [% retrieved_count %].
    </p>

    <ol>
    [% FOREACH document IN documents %]
        <li>
            <a href="/document/[% document.id %]">[% document.title %]</a>
        </li>
    [% END %]
    </ol>
    [% ELSE %]    
    <p>
        No hits -- try again!
    </p>
    [% END %]
[% END %]

This displays the phrase that was submitted, the number of hits, and a list of results (or a "no hits" message if there weren't any).

And you know what? We're now ready to actually do a search in the browser:

Neat, we have a working search application! We're just missing one important thing, and that is being able to access a document that was found. The results link to /document/:document_id, but that route isn't recognized by our app. No worries, we can fix that easily:

# Get the document with the specified ID
get '/document/:id' => sub {
    my $sth = database->prepare('SELECT contents_html FROM documents ' .
        'WHERE id = ?');
    $sth->execute(params->{'id'});
        
    if (my $document = $sth->fetchrow_hashref) {
        return $document->{'contents_html'};
    }
    else {
        status 404;
        return "Document not found";
    }
};

This route handler is pretty straightforward, we grab the ID from the URL, use it in a SELECT query to the documents table, and return the HTML contents of the matching document (or a 404 page, if there's no document with that ID).

The Complete Application

This is the complete DancerSearch.pm file:

package DancerSearch;

use Dancer 2.0;
use Dancer::Plugin::Database;
use HTML::Entities qw( encode_entities );
use Sphinx::Search;

# Create a new Sphinx::Search instance
my $sph = Sphinx::Search->new;
# Match all words, sort by relevance, return the first 10 results
$sph->SetMatchMode(SPH_MATCH_ALL);
$sph->SetSortMode(SPH_SORT_RELEVANCE);
$sph->SetLimits(0, 10);

get '/' => sub {
    if (my $phrase = params('query')->{'phrase'}) {
        # Send the search query to Sphinx
        my $results = $sph->Query($phrase);

        my $retrieved_count = 0;
        my $total_count;
        my $documents = [];

        if ($total_count = $results->{'total_found'}) {
            $retrieved_count = @{$results->{'matches'}};
            # Get the array of document IDs
            my @document_ids = map { $_->{'doc'} } @{$results->{'matches'}};
            # Join the IDs to use in SQL query (the IDs come from Sphinx, so we
            # can trust them to be safe)
            my $ids_joined = join ',', @document_ids;

            # Select documents, in the same order as returned by Sphinx
            my $sth = database->prepare('SELECT id, title FROM documents ' .
                "WHERE id IN ($ids_joined) ORDER BY FIELD(id, $ids_joined)");
            $sth->execute;

            # Fetch all results as an arrayref of hashrefs
            $documents = $sth->fetchall_arrayref({});
        }

        # Show search results page
        return template 'index', {
            phrase => encode_entities($phrase),
            retrieved_count => $retrieved_count,
            total_count => $total_count,
            documents => $documents
        };
    }
    else {
        # No search phrase -- show just the search form
        template 'index';
    }
};

# Get the document with the specified ID
get '/document/:id' => sub {
    my $sth = database->prepare('SELECT contents_html FROM documents ' .
        'WHERE id = ?');
    $sth->execute(params->{'id'});
        
    if (my $document = $sth->fetchrow_hashref) {
        return $document->{'contents_html'};
    }
    else {
        status 404;
        return "Document not found";
    }
};

1;

Conclusion

What we've built is still a very basic application, lacking many features -- the most obvious one that's missing is pagination, and being able to access results further down the list, not just the first ten. However, the code can be easily extended, thanks to the flexibility and ease of use of both Dancer and Sphinx. With a bit of effort, it can be made into an useful search app for a knowledge base site, or a wiki.

I think this application is a good example of how Dancer benefits from being part of the Perl ecosystem, giving web developers the ability to make use of the thousands of modules in CPAN (like we just did with Sphinx::Search). This allows to build working prototypes of web applications and implement complex features in a very short time.

The POD Extraction Script

As promised, this is the script that I used to extract the POD from Dancer distribution and store it in the MySQL database:

#!/usr/bin/env perl

package MyParser;

use strict;
use vars qw(@ISA);

use Pod::Simple::PullParser ();
BEGIN { @ISA = ('Pod::Simple::PullParser') }

use DBI;
use File::Find;
use Pod::Simple::Text;
use Pod::Simple::HTML;

# Variables to hold the text and HTML produced by POD parsers
my ($text, $html);
# Create parser objects and tell them where their output will go
(my $parser_text = Pod::Simple::Text->new)->output_string(\$text);
(my $parser_html = Pod::Simple::HTML->new)->output_string(\$html);

# Initialize database connection
my $dbh = DBI->connect("dbi:mysql:dbname=docs;host=localhost", "user",
    "hunter1")
    or die $!;

sub run {
    my $self = shift;
    my (@tokens, $title);

    while (my $token = $self->get_token) {
        push @tokens, $token;

        # We're looking for a "=head1 NAME" section
        if (@tokens > 5) {
            if ($tokens[0]->is_start && $tokens[0]->tagname eq 'head1' &&
                $tokens[1]->is_text && $tokens[1]->text =~ /^name$/i &&
                $tokens[4]->is_text)
            {
                $title = $tokens[4]->text;
                # We have the title, so we can ignore the remaining tokens
                last;
            }

            shift @tokens;
        }
    }

    # No title means no POD -- we're done with this file
    return if !$title;

    print "Adding: $title\n";

    $parser_text->parse_file($self->source_filename);
    $parser_html->parse_file($self->source_filename);

    # Add the new document to the database
    $dbh->do("INSERT INTO documents (title, contents_text, " .
        "contents_html) VALUES(?, ?, ?)", undef, $title, $text, $html);

    # Clear the content variables and reinitialize parsers
    $text = $html = "";
    $parser_text->reinit;
    $parser_html->reinit;
}

my $parser = MyParser->new;

find({ wanted => sub {
    if (-f and /\.pm$|\.pod$/) {
        $parser->parse_file($File::Find::name);
        $parser->reinit;
    }
}, no_chdir => 1 }, shift || '.');

You can run it with one argument, which is the location of the directory that will be scanned (recursively) for .pm/.pod files, or with no arguments, in which case the script will work with the current directory.

(Note: The script makes use of Pod::Simple, which I'm not very familiar with, so it's possible that I'm doing something stupid with it -- if that's the case, please let me know.)

This post was originally published as part of the 2012 Dancer Advent Calendar.

Template::Tiny is a lightweight engine which reimplements a subset of Template Toolkit features. As the name implies, it aims to accomplish this with as little code as possible. If you're using just the basic functionality of Template Toolkit, you should be able to switch to Template::Tiny without any modifications to template files (and you can easily go back at any moment).

Dancer::Template::Tiny is going to replace Dancer::Template::Simple as the default templating engine in Dancer2.

Example template:

<html>
  <head>
    <title>Tiny Example</title>
    <link rel="stylesheet" href="[% request.uri_base %]/css/style.css" />
  </head>
  <body>
    <h1>Hello, World! This is Dancer [% dancer_version %]!</h1>
    <p>
      [% IF morning %]
        Good morning!
      [% ELSE %]
        Good afternoon!
      [% END %] 
    </p>
  </body>
</html>

Route handler:

use DateTime;
    
get '/hello' => sub {
    template 'hello', { morning => (localtime)[2] < 12, now => DateTime->now };
};

Dancer::Template::Tenjin

Tenjin is a very fast templating engine with implementations for many languages -- including, of course, Perl. Its great performance comes from the fact that it uses the underlying language's constructs to process templates, instead of defining its own templating language and having to parse it. Support for this engine in Dancer is provided by Dancer::Template::Tenjin.

Example template:

<html>
  <head>
    <title>Tenjin Example</title>
    <link rel="stylesheet" href="[== $request->uri_base =]/css/style.css" />
  </head>
  <body>
    <h1>Hello, World! This is Dancer [= $dancer_version =]!</h1>
    <p>
      <?pl if ((localtime)[2] < 12) { ?>
        Good morning!
      <?pl } else { ?>
        Good afternoon!
      <?pl } ?>
    </p>
    <p>
      Current time is: [== DateTime->now->hms =]
    </p>
  </body>
</html>

Route handler:

use DateTime;
    
get '/hello' => sub {
    template 'hello';
};

Dancer::Template::Haml

Haml, which stands for "HTML Abstraction Markup Language", brings a fresh, different approach to templating. It aims at making templates short, clean, and as easy to read as well-formatted source code. Dancer::Template::Haml is a wrapper around Text::Haml and lets you use Haml templates in Dancer applications.

Example template:

%html
  %head
    %title Haml Example
    %link(rel="stylesheet" href="#{$request->uri_base}/css/style.css")
  %body
    %h1 Hello, World! This is Dancer #{$dancer_version}!
    %p
      - if ((localtime)[2] < 12) {
        Good morning!
      - } else {
        Good afternoon!
      - } 
    %p Current time is: #{DateTime->now->hms}

Route handler:

use DateTime;
    
get '/hello' => sub {
    template 'hello';
};

More

There are many more interesting templating engines ready to be used with Dancer, such as Mason (provided by Dancer::Template::Mason) or Xslate (Dancer::Template::Xslate). Do a CPAN or MetaCPAN search for "dancer template" to get a list of all the available engines, and choose the one that suits you best. In the true spirit of Perl, there's more than one way to write a template!

This post was originally published as part of the 2011 Dancer Advent Calendar.

I could just deploy the new Dancer website on Apache and keep using those goodies, but I thought that it would be nice if Dancer itself provided similar features. So, I created two plugins that do just that: Dancer::Plugin::DirectoryView and Dancer::Plugin::Auth::Htpasswd. Let me now show you how to use them.

Let's say we have a files directory under public, and we'd like to allow users to browse it and download files. Enabling directory access is as simple as including the plugin in our Dancer application:

      package MyWebApp;

      ...

      use Dancer::Plugin::DirectoryView;

And updating the configuration file (config.yml) to tell the plugin which directory should be made available, and at which URL:

      plugins:
          DirectoryView:
              url: /pub
              root_dir: files

That's it -- now, if we launch our app and point the browser at the /pub URL, we'll see the contents of the directory:

Protecting Directories with Htpasswd Files

As you might have noticed on the screenshot, there's a secret directory under files. It contains some super secret data that should only be available to authorized users, so now we're going to protect it using a htpasswd file.

First, let's create the htpasswd file and an user, named "alice":

      $ htpasswd -c htpasswd alice

Once it is created, we need to put the htpasswd file in a safe location outside of the public directory, so let's create a new directory passwd and store the file in there.

(If you're migrating from Apache and already have the htpasswd file, you just need to copy it to your Dancer application.)

In our Dancer application, we include the Auth::Htpasswd plugin:

      package MyWebApp;

      ...

      use Dancer::Plugin::Auth::Htpasswd;

Now, we need to update our configuration file and add settings for the plugin. We'll tell it to protect the /pub/secret path, and to use the htpasswd file we just created:

      plugins:
         "Auth::Htpasswd":
             paths:
                 "/pub/secret":
                     realm: "Secret Files"
                     passwd_file: passwd/htpasswd

The realm parameter lets us set the text that will be shown to the user in the login window displayed by the browser.

Let's see if our protection works. We restart the application and try to access the /pub/secret/ URL:

Great, our confidential files are safe. Only when we log in as "Alice", we'll be able to access them:

One of the features of GitHub is that it checks if a repository has a README file in its root directory, and displays it on the home page of the repository. This makes the README file a good place to introduce your project to the public.

GitHub also understands a number of markup languages, such as Markdown and Textile, and if the README file is in one of these formats, it will be transformed into nicely formatted HTML. One of the supported formats is POD, which means that the standard documentation of a Perl module can be used as its README file and serve as the repository's home page (much like on CPAN).

Module::Starter, which is Perl's recommended tool for building modules, does not create a GitHub-friendly README file -- instead, the README that it produces contains installation instructions (the "perl Makefile.PL; make..." mantra) and a couple links to module resources. This means that if you want to have a GitHub-friendly README file in your module, you need to either create it yourself, or tweak your build script a bit to have it generate it for you automatically.

This article wouldn't be particularly interesting if I told you to now go and make the README file yourself, would it? So let me show you how to do this automatically with ExtUtils::MakeMaker and Module::Build based modules (generated with Module::Starter). I will demonstrate how to create two README files: one being the POD version (named README.pod), the other one plain text (named just README).

ExtUtils::MakeMaker

Since ExtUtils::MakeMaker creates a Makefile with shell commands, you can tell it to generate the README files using two core Perl command-line utilities: perldoc (to generate POD from module's source) and pod2text (to convert POD into plain text). Extend Makefile.PL by adding the shell commands as the PREOP attribute of the dist target configuration:

my $preop =
    'perldoc -uT $(VERSION_FROM) | tee $(DISTVNAME)/README.pod > README.pod;' .
    'pod2text README.pod | tee $(DISTVNAME)/README > README';

WriteMakefile(
    NAME                => 'Foo::Bar',
    AUTHOR              => q{Michal Wojciechowski <odyniec@cpan.org>},
    VERSION_FROM        => 'lib/Foo/Bar.pm',
    ABSTRACT_FROM       => 'lib/Foo/Bar.pm',
    ($ExtUtils::MakeMaker::VERSION >= 6.3002
      ? ('LICENSE'=> 'perl')
      : ()),
    PL_FILES            => {},
    PREREQ_PM => {
        'Test::More' => 0,
    },
    dist                => {
        COMPRESS => 'gzip -9f',
        SUFFIX => 'gz',
        PREOP => $preop,
    },
    clean               => { FILES => 'Foo-Bar-*' },
);

Now, when you run perl Makefile.PL and make dist, the two README files will be created for you.

Don't worry if running make dist produces warnings that README and README.pod are missing -- it's no big deal, as the warnings will only be seen by you when making a distribution package, and not by the user building the module.

Module::Build

Module::Build defines a docs action, and it's the appropriate place for the code that builds the README files. Two modules that you can use for this purpose are Pod::Select and Pod::Readme. In your Build.PL file, create a subclass of Module::Build, and define a subroutine named ACTION_docs, similar to the one shown below:

my $class = Module::Build->subclass(
    class => 'My::Builder',
    code => q{
        sub ACTION_docs {
            use Pod::Readme;
            use Pod::Select;

            my $self = shift;

            podselect({ -output => 'README.pod' },
                'lib/Foo/Bar.pm');

            my $parser = Pod::Readme->new();
            $parser->parse_from_file('README.pod', 'README');

            return $self->SUPER::ACTION_docs;
        }
    }
);

my $builder = $class->new(
    module_name         => 'Foo::Bar',
    license             => 'perl',
    dist_author         => q{Michal Wojciechowski <odyniec@cpan.org>},
    dist_version_from   => 'lib/Foo/Bar.pm',
    requires => {
        ...
    },
    configure_requires => {
        'Pod::Readme' => 0,
        'Pod::Select' => 0,
    },
    build_requires => {
        'Pod::Readme' => 0,
        'Pod::Select' => 0,
        'Test::More' => 0,
    },
    add_to_cleanup      => [ 'Foo-Bar-*' ],
    create_makefile_pl => 'traditional',
);

$builder->create_build_script();

You can now run perl Build.PL, and then ./Build docs, to build the README files.

Remember to add the Pod::Select and Pod::Readme modules to configure_requires and build_requires, as shown in the above example.

Installation instructions

Since these methods overwrite the original README file provided by Module::Starter, the installation instructions in it are also lost. It's a good practice to always include installation instructions, so go ahead and add an INSTALL file to your module's distribution files. It can be really simple and straight to the point:

Foo-Bar

INSTALLATION

To install this module, run the following commands:

    perl Build.PL
    ./Build
    ./Build test
    ./Build install

Finally, remember to add README.pod and INSTALL to your module's MANIFEST.