Using Minion with a REST API

Minion is a job queue for the Mojolicious real-time web framework. Below is example usage that sends an email using a REST API. The email address is handed off to minion and then minion processes the task. Everything is in a self-contained file as a Mojolicious::Lite app.

In addition, a config file is shared between minion and the web app.

One cool thing about Minion is that it has support for a Postgres backend using Mojo::Pg. Given that, workers can be running jobs on different physical boxes as long as they use a compatible "Pg" connect string.

The web app is started like so:

$ /opt/perl minion_email.pl daemon

A Minion worker is started with:

$ /opt/perl minion_email.pl minion worker

Example usage:

$ curl -s -X POST http://localhost:3000/api/v1/stats -d '{"api_key":"68b329da9893e34099c7d8ad5cb9c940","username"}' | ~/jq . -                       
{
  "status": "success",
  "message": {
    "inactive_workers": 1,
    "finished_jobs": "41",
    "active_jobs": 0,
    "failed_jobs": 0,
    "active_workers": 0,
    "inactive_jobs": 0
  }
}
$ curl -s -X POST http://localhost:3000/api/v1/email -d '{"api_key":"68b329da9893e34099c7d8ad5cb9c940","username":"fnord","email":"joy@example.com"}' | ~/jq . -
{
  "jobid": "139",
  "message": "Email send to joy@example.com",
  "status": "success"
}
$ curl -s -X POST http://localhost:3000/api/v1/status -d '{"api_key":"68b329da9893e34099c7d8ad5cb9c940","username":"fnord","jobid":"139"}' | ~/jq . -               
{
  "status": "success",
  "message": {
    "retries": 0,
    "retried": null,
    "args": [
      "joy@example.com"
    ],
    "result": {
      "status": "success",
      "msg": "Mail to joy@example.com sent"
    },
    "state": "finished",
    "worker": 669,
    "id": "139",
    "priority": 0,
    "delayed": "1",
    "finished": "1415590829.85986",
    "started": "1415590829.30462",
    "created": "1415590825.31839",
    "task": "email"
  }
}
$ curl -s -X POST http://localhost:3000/api/v1/stats -d '{"api_key":"68b329da9893e34099c7d8ad5cb9c940","username":"fnord"}' | ~/jq . -
{
  "status": "success",
  "message": {
    "finished_jobs": "42",
    "inactive_workers": 1,
    "inactive_jobs": 0,
    "failed_jobs": 0,
    "active_workers": 0,
    "active_jobs": 0
  }
}
$ cat /opt/minion_email/config
{
    smtp_host => "*",
    smtp_port => '*',
    smtp_user => '*',
    smtp_pass => '*',
    pg_string => 'postgresql://username:password@localhost/jobs',
    secret => '*',
};

Below is the Mojolicious::Lite example script:

use Mojolicious::Lite;

use v5.20;
use experimental 'signatures';

use Email::Sender::Simple qw(sendmail);
use Email::Simple;
use Email::Simple::Creator;
use Email::Sender::Transport::SMTP::TLS;
use Email::Valid;

use Mojo::JSON 'encode_json';

plugin Config => {file => '/opt/minion_email/config'};
plugin Minion => {Pg => app->config->{pg_string}};

app->secrets([app->config->{secret}]);

app->minion->add_task(email => sub ($job, $email) {
    my $mail = Email::Simple->create(
        header => [
            To     => $email,
            From    => 'signup@host.com',
            Subject => "Super Signup Email",
        ],
        body => "Thank you for signing up with us.\nGlobal conquest in an hour\n",
    );

    my $transport = Email::Sender::Transport::SMTP::TLS->new({
            host => app->config->{smtp_host},
            port => app->config->{smtp_port},
            username => app->config->{smtp_user},
            password => app->config->{smtp_pass},
            timeout => 10,
    });

    eval {
        sendmail($mail, {transport => $transport });
    };
    if ($@) {
        $job->app->log->debug("error: $@");
        $job->finish({ status => "error", msg => $@});
    }
    else {
        $job->finish({ status => "success", msg => "Mail to $email sent"});
    }
});

get '/' => 'index';

under (sub {
    my $self = shift;

    return($self->render(json => {status => "error", data => { message => "No JSON found" }})) unless $self->req->json;

    my $username = $self->req->json->{username};
    my $api_key = $self->req->json->{api_key};

    unless ($username) {
        $self->render(json => {status => "error", data => { message => "No username found" }});

        return undef;
    }

    unless ($api_key) {
        $self->render(json => {status => "error", data => { message => "No API Key found" }});

        return undef;
    }

    unless ("fnord" eq $username) {
        $self->render(json => {status => "error", data => { message => "Credentials mis-match" }});

        return undef;
    }

    unless ("68b329da9893e34099c7d8ad5cb9c940" eq $api_key) {
        $self->render(json => {status => "error", data => { message => "Credentials mis-match" }});

        return undef;
    }

    return 1;
});

post '/api/v1/email' => sub ($c) {
    my $email = $c->req->json->{email};

    unless (Email::Valid->address($email)) {
        return($c->render(json => {status => "error", message => "Email does not seem to be valid"}));
    }

    my $jobid = $c->minion->enqueue(email => [$email]);

    return($c->render(json => {status => "success", message => "Email send to $email", jobid => $jobid}));
};

post '/api/v1/status' => sub ($c) {
    my $jobid = $c->req->json->{jobid};

    unless ($jobid =~ m/^\d+$/) {
        return($c->render(json => {status => "error", message => "JobID not an unsigned integer"}));
    }

    my $job = $c->minion->job($jobid);
    if ($job) {
        return($c->render(json => {status => "success", message => $job->info}));
    }
    else {
        return($c->render(json => {status => "error", message => "Job not: found for $jobid"}));
    }
};

post '/api/v1/stats' => sub ($c) {
    my $stats = $c->minion->stats;
    return($c->render(json => {status => "success", message => $stats}));
};

app->start;

__DATA__

@@ index.html.ep

Try:<br>

    POST /email/v1/email
    POST /email/v1/status

Running a non-blocking command in Mojolicious

Mojolicious is an awesome framework designed from the ground up to be non-blocking. During web development it is sometimes desirable to run a command and do something with the results. However, with methods such as "system" and "open" the command will block concurrent requests.

A non-blocking process execution is possible with background processes; however, the process must be managed and if the UserAgent needs updating, then long-polling will need to be utilized.

Below is a complete Mojolicious::Lite program that does this. In words, the app generates a utility script on-the-fly that a) sleeps; b) writes to a state file; and c) sleeps at the the end. This means that our state file will be around while the program is running. This mimics several use-cases, for example creating a thumbnail.

The state file can be inspected using events provided by Mojo::IOLoop::ProcBackground. The alive event is a heartbeat for the process and the dead event happens when the process stops.


my $proc = $self->stash->{_proc} = Mojo::IOLoop::ProcBackground->new;

# Every so often we get a heartbeat from the background process
$proc->on(alive => sub {
    my ($proc) = @_;

    # ...
});

# When the process terminates, we get this event
$proc->on(dead => sub {
    my ($proc) = @_;

    # ...

    $self->finish;   # The GET request is done now
});

# Start our process
$proc->run([$^X, $script, $statefile]);
Here is the full example:
C:\>perl poll.pl
[Thu Feb 13 22:37:51 2014] [info] Listening at "http://*:5555".
Server available at http://127.0.0.1:5555.
[Thu Feb 13 22:37:55 2014] [debug] GET "/run".
[Thu Feb 13 22:37:55 2014] [debug] Routing to a callback.
[Thu Feb 13 22:37:55 2014] [debug] 200 OK (0.001246s, 802.568/s).
[Thu Feb 13 22:38:08 2014] [debug] Done: C:\DOCUME~1\ADMINI~1\LOCALS~1\Temp\done.3676
[Thu Feb 13 22:38:08 2014] [debug] Finished

...

C:\Documents and Settings\Administrator>mojo get "http://127.0.0.1:5555/run"
Starting...
Done C:\Documents and Settings\Administrator>

The code was tested in OS X and Win XP.


use Mojolicious::Lite;

use Mojo::IOLoop::ProcBackground;

use File::Temp;
use File::Spec;
use Proc::Background;

any '/run' => sub {
        my $self = shift;

        # Setup our request to take a while
        Mojo::IOLoop->stream($self->tx->connection)->timeout(30);
        $self->render_later;

        $self->on(finish => sub {
            $self->app->log->debug("Finished");
        });

        # We want the UserAgent to see something as soon as possible
        $self->res->code(200);
        $self->res->headers->content_type('text/html');
        $self->write_chunk("Starting...
\n"); # This is our utility script that will run in the background my $tmp = File::Temp->new(UNLINK => 0, SUFFIX => '.pl'); my $statefile = $self->stash->{_statefile} = File::Spec->catfile(File::Spec->tmpdir, "done"); print($tmp 'sleep(10); $f="$ARGV[0].$$"; open($fh, ">", $f); sleep(3)'); my $script = $tmp->filename; undef($tmp); # Thanks CPAN.. :) The magic happens in Proc::Background my $proc = $self->stash->{_proc} = Mojo::IOLoop::ProcBackground->new; # Every so often we get a heartbeat from the background process $proc->on(alive => sub { my ($proc) = @_; my $pid = $proc->proc->pid; my $statefile = $self->stash->{_statefile} . ".$pid"; if (-f $statefile) { $self->write_chunk("Done"); $proc->unsubscribe("alive"); } }); # When the process terminates, we get this event $proc->on(dead => sub { my ($proc) = @_; my $pid = $proc->proc->pid; my $statefile = $self->stash->{_statefile} . ".$pid"; $self->app->log->debug("Done: $statefile"); $self->finish; }); # Start our process $proc->run([$^X, $script, $statefile]); }; # Run the app push(@ARGV, 'daemon', '-l', 'http://*:5555') unless @ARGV; app->log->level("debug"); app->secrets(["I Knos# you!!"]); app->start;

It should be noted that there is more than one way to run a process. For example, Mojo::IOLoop::ReadWriteFork can be used to follow the process STDOUT/STDERR; and Mojo::IOLoop::ProcBackground simply runs a command in the background and allows for job control; and Mojo::IOLoop::ForkCall can be used to execute arbitrary perl in an async fashion (you can run a command, as well) and have programmatic control over the output.

In summary, one allows for STDOUT/STDERR events; one simply runs a command; and another allows for arbitrary perl to be utilized.

Fetch me a cloud

At times, it is desirable to access any file on your box. Forgot to put it on dropbox? Then what do you do? Enter FileBeagle. It is your own personal cloud and written in Perl and Mojolicious. The install is copying one file onto your box and running the executable. The only other file that is permanent is the database that is created.

There is currently a Lite edition with a Pro version coming soon. The Lite version is free.

Announce CloseBargains.com

It has been said that we need more Perl startups. I agree and have written CloseBargains.com - what better way to get a heapin helpin of local coupons than Perl, Mojolicious, Linux, and some 8coupons API goodness served up by hypnotoad. The frontend is done in jQuery Mobile; another buzzword that needs adding is that Backbone.js needs to be integrated for coolness factor.

Headless Selenium testing with PhantomJS

As you know, Selenium is a marvelous library for automating a browser. It can be combined with Test::More and PhantomJS to provide a headless test suite.

An example script looks like:


#!/opt/perl

use Selenium::Remote::Driver;
use Selenium::Remote::WDKeys;

use Test::More;
use MIME::Base64;

my $driver = Selenium::Remote::Driver->new();

$driver->set_implicit_wait_timeout(3000);

$driver->get('http://www.google.com');
like($driver->get_title(), qr/^Google$/, "Arrived at Google homepage");

my $elem = $driver->find_element("input[name=q]", "css");
$elem->send_keys("Mojolicious");

open(my $fh,'>','mojoText.png');
binmode($fh);
my $png_base64 = $driver->screenshot();
print($fh MIME::Base64::decode_base64($png_base64));
close($fh);

$elem->send_keys(KEYS->{'enter'});

$elem = $driver->find_element("#resultStats", "css");
like($elem->get_text(), qr/About.*results/, "Got some results");

open($fh,'>','mojoResults.png');
binmode($fh);
my $png_base64 = $driver->screenshot();
print($fh MIME::Base64::decode_base64($png_base64));
close($fh);

$driver->quit();

done_testing();

This works under the assumption that PhantomJS is running like so:

[bpm@s001] c:~>./phantomjs --webdriver=4444
PhantomJS is launching GhostDriver...
Ghost Driver running on port 4444

The output looks like:

[bpm@s001] c:~/playground>/opt/perl google.pl 
ok 1 - Arrived at Google homepage
ok 2 - Got some results
1..2
[bpm@s001] c:~/playground>