Using Minion with a REST API

Minion is a job queue for the Mojolicious real-time web framework. Below is example usage that sends an email using a REST API. The email address is handed off to minion and then minion processes the task. Everything is in a self-contained file as a Mojolicious::Lite app.

In addition, a config file is shared between minion and the web app.

One cool thing about Minion is that it has support for a Postgres backend using Mojo::Pg. Given that, workers can be running jobs on different physical boxes as long as they use a compatible "Pg" connect string.

The web app is started like so:

$ /opt/perl minion_email.pl daemon

A Minion worker is started with:

$ /opt/perl minion_email.pl minion worker

Example usage:

$ curl -s -X POST http://localhost:3000/api/v1/stats -d '{"api_key":"68b329da9893e34099c7d8ad5cb9c940","username"}' | ~/jq . -                       
{
  "status": "success",
  "message": {
    "inactive_workers": 1,
    "finished_jobs": "41",
    "active_jobs": 0,
    "failed_jobs": 0,
    "active_workers": 0,
    "inactive_jobs": 0
  }
}
$ curl -s -X POST http://localhost:3000/api/v1/email -d '{"api_key":"68b329da9893e34099c7d8ad5cb9c940","username":"fnord","email":"joy@example.com"}' | ~/jq . -
{
  "jobid": "139",
  "message": "Email send to joy@example.com",
  "status": "success"
}
$ curl -s -X POST http://localhost:3000/api/v1/status -d '{"api_key":"68b329da9893e34099c7d8ad5cb9c940","username":"fnord","jobid":"139"}' | ~/jq . -               
{
  "status": "success",
  "message": {
    "retries": 0,
    "retried": null,
    "args": [
      "joy@example.com"
    ],
    "result": {
      "status": "success",
      "msg": "Mail to joy@example.com sent"
    },
    "state": "finished",
    "worker": 669,
    "id": "139",
    "priority": 0,
    "delayed": "1",
    "finished": "1415590829.85986",
    "started": "1415590829.30462",
    "created": "1415590825.31839",
    "task": "email"
  }
}
$ curl -s -X POST http://localhost:3000/api/v1/stats -d '{"api_key":"68b329da9893e34099c7d8ad5cb9c940","username":"fnord"}' | ~/jq . -
{
  "status": "success",
  "message": {
    "finished_jobs": "42",
    "inactive_workers": 1,
    "inactive_jobs": 0,
    "failed_jobs": 0,
    "active_workers": 0,
    "active_jobs": 0
  }
}
$ cat /opt/minion_email/config
{
    smtp_host => "*",
    smtp_port => '*',
    smtp_user => '*',
    smtp_pass => '*',
    pg_string => 'postgresql://username:password@localhost/jobs',
    secret => '*',
};

Below is the Mojolicious::Lite example script:

use Mojolicious::Lite;

use v5.20;
use experimental 'signatures';

use Email::Sender::Simple qw(sendmail);
use Email::Simple;
use Email::Simple::Creator;
use Email::Sender::Transport::SMTP::TLS;
use Email::Valid;

use Mojo::JSON 'encode_json';

plugin Config => {file => '/opt/minion_email/config'};
plugin Minion => {Pg => app->config->{pg_string}};

app->secrets([app->config->{secret}]);

app->minion->add_task(email => sub ($job, $email) {
    my $mail = Email::Simple->create(
        header => [
            To     => $email,
            From    => 'signup@host.com',
            Subject => "Super Signup Email",
        ],
        body => "Thank you for signing up with us.\nGlobal conquest in an hour\n",
    );

    my $transport = Email::Sender::Transport::SMTP::TLS->new({
            host => app->config->{smtp_host},
            port => app->config->{smtp_port},
            username => app->config->{smtp_user},
            password => app->config->{smtp_pass},
            timeout => 10,
    });

    eval {
        sendmail($mail, {transport => $transport });
    };
    if ($@) {
        $job->app->log->debug("error: $@");
        $job->finish({ status => "error", msg => $@});
    }
    else {
        $job->finish({ status => "success", msg => "Mail to $email sent"});
    }
});

get '/' => 'index';

under (sub {
    my $self = shift;

    return($self->render(json => {status => "error", data => { message => "No JSON found" }})) unless $self->req->json;

    my $username = $self->req->json->{username};
    my $api_key = $self->req->json->{api_key};

    unless ($username) {
        $self->render(json => {status => "error", data => { message => "No username found" }});

        return undef;
    }

    unless ($api_key) {
        $self->render(json => {status => "error", data => { message => "No API Key found" }});

        return undef;
    }

    unless ("fnord" eq $username) {
        $self->render(json => {status => "error", data => { message => "Credentials mis-match" }});

        return undef;
    }

    unless ("68b329da9893e34099c7d8ad5cb9c940" eq $api_key) {
        $self->render(json => {status => "error", data => { message => "Credentials mis-match" }});

        return undef;
    }

    return 1;
});

post '/api/v1/email' => sub ($c) {
    my $email = $c->req->json->{email};

    unless (Email::Valid->address($email)) {
        return($c->render(json => {status => "error", message => "Email does not seem to be valid"}));
    }

    my $jobid = $c->minion->enqueue(email => [$email]);

    return($c->render(json => {status => "success", message => "Email send to $email", jobid => $jobid}));
};

post '/api/v1/status' => sub ($c) {
    my $jobid = $c->req->json->{jobid};

    unless ($jobid =~ m/^\d+$/) {
        return($c->render(json => {status => "error", message => "JobID not an unsigned integer"}));
    }

    my $job = $c->minion->job($jobid);
    if ($job) {
        return($c->render(json => {status => "success", message => $job->info}));
    }
    else {
        return($c->render(json => {status => "error", message => "Job not: found for $jobid"}));
    }
};

post '/api/v1/stats' => sub ($c) {
    my $stats = $c->minion->stats;
    return($c->render(json => {status => "success", message => $stats}));
};

app->start;

__DATA__

@@ index.html.ep

Try:<br>

    POST /email/v1/email
    POST /email/v1/status

4 Comments

That's not REST. I immediately noticed several huge code/design smells:

  1. Authentication is tunnelled through the body. Remedy: use a standard authentication scheme that works with HTTP headers or TLS.
  2. Status is tunnelled through the body. Remedy: return the appropriate 4xx header for client errors.
  3. There are no hyperlinks between resources. Remedy: add hypermedia controls, e.g. by upgrading to HAL+JSON.
  4. A client cannot easily traverse the URI space from the entry point. Remedy: define link relations and add them to hyperlinks, add the OPTIONS method and Allow, Accept-Post headers.
  5. Versioning couples clients tightly to the implementation, guaranteeing breakage when you change your code. Remedy: add a layer of indirection and shift the contract from magic URI substrings towards link relations. Mutate the interface piecemeal and in phases.

Thanks fmtyew.tk, "your ideas are intriguing to me and I wish to subscribe to your newsletter." The one I struggle with in your list is the Versioning. Can you elaborate on this a little more? FYI We use Catalyst and I'm not confident on a good approach for versioning.

When you want to change the semantics of a resource, its type changes, so coin a new link relation for it. You can link to resources of both old and new type for a transitional period in parallel, and mark the old one as deprecated. You do so by amending the document a client gets when dereferencing a link relation URI. HAL even has built-in deprecation support on the link itself.

When appropriate, you can employ redirects with 301 or 303, it depends on the concrete semantics of a resource, e.g. whether it's backward compatible. When you notice in your logs that most clients do not follow links of deprecated type anymore, you can retire those resources with 410 and remove the links. That's versioning in a nutshell. There are no hard cut-offs with names like v1, etc. but a manageable flux of changes over time, just like the Web itself.

Leave a comment

About Brian Medley

user-pic I blog about Perl.