Examples Archives

Non-blocking Mojolicious apps are even easier now!

Hopefully by now you have seen that Mojolicious is a great way to build your non-blocking web (or even non-web) application. Mojolicious come with all kinds of non-blocking functionality out of the box. For more about that see my blog series on the topic. This post is an aside to show you the cool things happening in Mojolicious lately designed to make writing non-blocking apps easier.

Mojolicious is known for fast development and clean APIs. Mojo was that child with lots of excitement and energy, doing new and cool things, providing new and cool functionality, and yes, changing its mind on occasion. But Mojo is growing up and settling down a little bit. It recently went to its first conference and professional training. And it’s starting a family too!

Mojo is starting to feel more grown up, and grown-ups have responsibilities. To borrow one of Perl’s catch phrases, this more mature Mojo knows that it is not good enough anymore to just make things possible, it’s time to make them easy.

Writing Non-Blocking Applications with Mojolicious: Part 3

This the the third part in an on-going series about using Mojolicious to write non-blockging applications (with an eye towards the web, obviously). In part 1 I demonstrated the how it can improve the number of requests/clients served when the application uses high-latency backends (in that case a database). In part 2 I showed how each request can be sped up when that request needs multiple resources from a high-latency service (e.g. external web services).

In each, I showed a blocking example, then a non-blocking example. I then gave the usual warning that you had to use a Mojolicious server for the nonblocking version. While its true that you need a Mojolicious server to get the benefits of the nonblocking architecture, in this post I will show how with a little care in construction, you can build your application so that it will run correctly on any supported server and the nonblocking benefits will be evident where possible.

Writing Non-Blocking Applications with Mojolicious: Part 2

Last time, I showed you how to write non-blocking (web) applications using Mojolicious. In those examples, each action only had to perform one non-blocking action. In this article I’m going to take things a little further, introducing you to Mojo::IOLoop::Delay. I will use a delay object to perform multiple simultaneous non-blocking actions and wait until they complete before moving on, all without blocking the server for other requests.

Writing Non-Blocking Applications with Mojolicious: Part 1

One question I often hear is “Why should I chose Mojolicious versus one of the other major Perl web frameworks?” While I have many answers to that question, I personally believe the most important difference is that Mojolicious is designed to be non-blocking. Many of you will have heard of Node.js. The reason that Node is popular is that it is designed to be non-blocking. By writing your webapp in a non-blocking style using a non-blocking framework, you can often build a faster and smarter application, requiring fewer servers to handle the same amount of traffic. Although Perl has several web frameworks, only one was written with non-blocking design in mind: Mojolicious.

To demonstrate a non-blocking application, I am going to write a simple pastebin using Mojolcious and Mango, a non-blocking MongoDB library (from the same developers as Mojo).

The Templates

Before I dive into the server-side code, let’s look at the templates which will form the view of the application. Mojolicious has its own templating engine which is a thin layer over normal Perl syntax.

@@ layouts/basic.html.ep

<!DOCTYPE html>
<html>
  <head>
    <title><%= title =%></title>
    %= stylesheet '//netdna.bootstrapcdn.com/bootstrap/3.0.0/css/bootstrap.min.css'
  </head>
  <body>
    <div class="container">
      <%= content =%>
    </div>
  </body>
</html>

@@ show.html.ep

% title $doc->{title};
% layout 'basic';

%= stylesheet begin
pre.prettyprint {
  background-color:inherit;
  border:none;
}
% end
%= tag h1 => $doc->{title}
%= tag div => class => 'well' => begin
  %= tag pre => class => 'prettyprint' => begin
    <%= $doc->{content} =%>
  % end
% end
%= javascript 'https://google-code-prettify.googlecode.com/svn/loader/run_prettify.js'

@@ submit.html.ep

% title 'Paste your content';
% layout 'basic';

%= form_for '/' => role => form => method => POST => begin
  %= tag div => class => 'form-group' => begin
    %= tag label => for => 'title' => 'Title'
    %= text_field 'title', class => 'form-control'
  % end
  %= tag div => class => 'form-group' => begin
    %= tag label => for => 'content' => 'Paste Content'
    %= text_area  'content', class => 'form-control'
  % end
  %= submit_button 'Paste' => class => 'btn btn-primary'
% end

The first template defines a basic html layout which will be applied to any other template that requests it. The contents of the requesting template will be inserted at the content directive. The show and submit templates (both of which request the basic layout) are the views that will be seen when a user wants to display a paste or create new one. Notice that in the show template, a variable $doc is seen. This (template lexical) variable will be created by the controller using the stash command later. Any functions you see are called helpers in Mojolicious parlance and are more accurately curried controller methods. Some style is applied by using Twitter’s Bootstrap and Google’s Prettify from CDN sites.

These templates will be used in both versions of my following code; they may be inserted as-is in the __DATA__ section of the following files or else placed separately in a directory named templates.

The Blocking Form

Since most people are comfortable with writing web applications using a blocking style, I will first present my paste application in this way.

#!/usr/bin/env perl

use Mojolicious::Lite;
use Mango;
use Mango::BSON 'bson_oid';

helper mango  => sub { state $mango = Mango->new($ENV{PASTEDB}) };
helper pastes => sub { shift->mango->db->collection('pastes') };

get '/' => 'submit';

post '/' => sub {
  my $self = shift;
  my $title = $self->param('title') || 'Untitled';
  my $content = $self->param('content')
    or return $self->redirect_to('/');
  my $doc = {
    title   => $title,
    content => $content,
  };
  my $oid = $self->pastes->save($doc);
  $self->redirect_to( show => id => "$oid" ); 
};

get '/:id' => sub {
  my $self = shift;
  my $id = bson_oid $self->stash('id');
  my $doc = $self->pastes->find_one({ _id => $id })
    or return $self->redirect_to('/');
  $self->stash( doc => $doc );
} => 'show';

app->start;

After importing the necessary libraries (importing Mojo turns on strict, warnings and utf8 and all v5.10 features), I build some helpers of my own. The mango helper will connect to a Mongo instance whose uri I will specify in an environment variable (yes I could put it in a configuration file, but I had to draw the line on this example somewhere :-) ). I have a helper which will return an instance of the collection (read: table) which will store the paste information. By default Mango will generate a unique document ID, which for our purposes, we will use as our page identifier.

Once the helpers have been created, three routes are defined and connected to controller callbacks (the lite version of controller methods), whose purposes should be rather self explanatory. One quick thing I note is that since I don’t explicitly called render, the controller will automatically render the template with the same name, where the name is defined in lite syntax by a string after the controller callback. I could have as used the render method, but this is more concise.

Running the application (seen in its completeness here, don’t forget the database env var!) will start the server. This application may be run as-is under CGI, any PSGI server, or using Mojolicious’ built-in servers. The application should run as you expect, however it has a major drawback! Any time any client causes the app to make a request to the database, all clients must wait for it to respond before the server may serve the next client. Meanwhile the server is sitting idle waiting for a response. Seems inefficient doesn’t it?!

The Non-Blocking Form

I can now present a very similar application, but which has a couple tweaks to prevent database calls from blocking the application.

#!/usr/bin/env perl

use Mojolicious::Lite;
use Mango;
use Mango::BSON 'bson_oid';

helper mango  => sub { state $mango = Mango->new($ENV{PASTEDB}) };
helper pastes => sub { shift->mango->db->collection('pastes') };

get '/' => 'submit';

post '/' => sub {
  my $self = shift;
  my $title = $self->param('title') || 'Untitled';
  my $content = $self->param('content')
    or return $self->redirect_to('/');
  my $doc = {
    title   => $title,
    content => $content,
  };
  $self->render_later;
  $self->pastes->save($doc, sub {
    my ($coll, $err, $oid) = @_;
    $self->redirect_to( show => id => "$oid" ); 
  });
};

get '/:id' => sub {
  my $self = shift;
  my $id = bson_oid $self->stash('id');
  $self->render_later;
  $self->pastes->find_one({ _id => $id }, sub {
    my ($coll, $err, $doc) = @_;
    return $self->redirect_to('/') if ( $err or not $doc );
    $self->render( show => doc => $doc );
  });
} => 'show';

app->start;

This new code only differs by a few lines, but they are important ones! First, before I make a non-blocking call, I need to call render_later to prevent the automatic rendering I mentioned above; the server cannot render when it reaches the end of the action method because it doesn’t have the data yet!

    title   => $title,
    content => $content,
  };
-  my $oid = $self->pastes->save($doc);
-  $self->redirect_to( show => id => "$oid" ); 
+  $self->render_later;
+  $self->pastes->save($doc, sub {
+    my ($coll, $err, $oid) = @_;
+    $self->redirect_to( show => id => "$oid" ); 
+  });
};

In the POST handler, you can see that I call the same save method on the collection, however, I don’t just pass the document, I also pass a subroutine reference (a callback) which will get called when the database finishes inserting the data. When a client posts the new paste to the server, it will wait until the database finishes storing the document before being redirected to view the document. Unlike the preview version of the code however, while waiting to serve this client, the server is able to move on to serve other clients which are waiting for other data!

get '/:id' => sub {
  my $self = shift;
  my $id = bson_oid $self->stash('id');
-  my $doc = $self->pastes->find_one({ _id => $id })
-    or return $self->redirect_to('/');
-  $self->stash( doc => $doc );
+  $self->render_later;
+  $self->pastes->find_one({ _id => $id }, sub {
+    my ($coll, $err, $doc) = @_;
+    return $self->redirect_to('/') if ( $err or not $doc );
+    $self->render( show => doc => $doc );
+  });
} => 'show';

Similar changes are made to the show controller. Once again, I call render_later and then move my post-database call logic into a callback. This time, the server will make a request of the database to get the relevant document, but then proceeds to serve other clients until the database responds. When the response comes back, the server invokes the callback and the client sees the requested page.

One small drawback is that now you must deploy using Mojolicious’ built-in servers. No need to fear however, they are very capable.

Ok this is cool but …

Yes it seem like a little more code, but the payoff comes next. If you are running your database on the same computer as the server, you will see little or no difference in performance. However, perhaps you want to use an off-site database host like MongoHQ. Using the tool wrk you can clearly see the benefit of rewriting your blocking application:

$ PASTEDB=mongodb://demo:pass@linus.mongohq.com:10025/MangoTest hypnotoad blocking_paste.pl
$ wrk -t 10 -c 10 -d 1m http://localhost:8080/0
Running 1m test @ http://localhost:8080/0
  10 threads and 10 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   286.87ms  331.49ms   1.45s    90.25%
    Req/Sec     5.91      4.32    22.00     81.12%
  3745 requests in 1.00m, 2.83MB read
Requests/sec:     62.41
Transfer/sec:     48.32KB
$ PASTEDB=mongodb://demo:pass@linus.mongohq.com:10025/MangoTest hypnotoad -s blocking_paste.pl

… in a non-blocking style:

$ PASTEDB=mongodb://demo:pass@linus.mongohq.com:10025/MangoTest hypnotoad nonblocking_paste.pl
$ wrk -t 10 -c 10 -d 1m http://localhost:8080/0
Running 1m test @ http://localhost:8080/0
  10 threads and 10 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    59.41ms   18.69ms 365.66ms   97.45%
    Req/Sec    16.73      2.19    21.00     75.56%
  10290 requests in 1.00m, 7.78MB read
Requests/sec:    171.49
Transfer/sec:    132.77KB
$ PASTEDB=mongodb://demo:pass@linus.mongohq.com:10025/MangoTest hypnotoad -s nonblocking_paste.pl

Please keep in mind, this is only a toy case on a free database and an older computer. That said, it is clear that there is a marked improvement in transfer and requests handled. Try it yourself, the code for both scripts is available here.

Conclusion

Mojolicious is a very powerful web framework which makes hard things easy. This is especially true for non-blocking code. I hope this post is the first in a series on writing non-blocking webapps (and perhaps even more general apps), using Mojolicious. Happy Perling!

P.S. see Mojo::IOLoop::Delay for even more ways that the Mojolicious tool suite (as I like to call it) makes writing non-blocking code easier.

This post is the first in an on-going series, continued in part 2

Some code ports to Mojolicious, just for fun.

Today is a relatively minor holiday in the US, but I had work off all the same. I found myself experimenting (when I probably should have been working on my YAPC::Brazil talk :-/). Thanks to today’s PerlWeekly (you are a subscriber right??), I found out about an interesting post by Johnny Moreno which creates a tiny json service backend using Perl. Of course it uses CGI.pm to do so, which made me curious what a Mojolcious port would look like.

My "Mojolicious Introduction" now updated for 4.0

On Feb 28, 2013 I gave a talk to Chicago.pm about Mojolicious. I called it an introduction, but I really wanted to show some of the features that sets Mojolicious apart. Because of this, the talk moves very fast. It hits routing and responses quickly, hits testing often, on all the way to well-tested non-blocking websocket examples.

I promised to get my slides up afterwards but life (i.e. my doctoral thesis) got in the way. Now with the release of Mojolicious 4.0 I thought I would take the opportunity to right a wrong and get the slides up; so here they are: http://mojolicious-introduction.herokuapp.com/!

The talk is itself a Mojolicious app, the source of which is available from on GitHub. Not only are all the code snippets shown in the talk included, not only do they all run, but they are actually what is rendered by the talk (DRY++), so what you see is what you get! Please leave any feedback and ask any questions. I may not see the responses here, so feel free to ping me elsewhere if needed.

Thank you Ack!

People may have noticed my absence from the Perl world lately. I have been writing my Ph.D. thesis (179 pages on Ultrafast Electron Microscopy with my Physics::UEMColumn Perl module featured) and defense.

Ack is a tool for searching code and text. It works much like the unix tool grep, although it is imbued with the power of Perl. To mark the release of Ack 2.0 though I wanted to mention a few one-liners that made my life easier in this stressful time.

My thesis is written in many LaTeX files and one can probably imagine that searching those files was needed regularly. The biggest is for finding non-ascii characters. As I add content from old publications or external programs, lots of non-ascii characters can often come along for the ride. LaTeX is a very old program, well pre-dating unicode, and it has a very different way of adding special characters with its own markup. In fact parts of the compiling toolchain croak with unicode characters. So I found myself using

ack '[^[:ascii:]]'

regularly. Also I had to keep a list of all the abbreviations that I had used in the paper, but of course you forget if you have them all. I used this little bash-ack conglomeration to find all sequences of two or more upper-case characters, which is how I write my abbreviations,

ack -ho '\p{Upper}{2,}' | sort | uniq

I’m sure I used many other little ackings here and there, but these were the two I could remember off-hand. Thanks to Andy and everyone who has contributed to ack!

Now go use it to make your life easier!

Visit the new site: beyondgrep.com

A WebSocket Mojolicious/DBI Example

Building on my (second) example from my recent post A Simple Mojolicious/DBI Example, I thought I would take it just a little bit futher. One of my favorite features of Mojolicious is that it comes with WebSockets out of the box! In this example I show how you can take the example script and allow it to run without refreshing the window.

A Simple Mojolicious/DBI Example

A friend of mine is learning database administration, but for the first time needs to connect it to a webapp. Having done most of his work in MS Access, he has no idea what a web framework is, nor how one would connect a database to it. To help him learn I wrote up a little application using Mojolicious and SQLite.

Having fun with some modern web technologies

Edit: MojoCMS has been renamed to Galileo and released to CPAN. Enjoy!

Over the holiday break, I decided to have a little fun learning some things about the web. I usually get my Perl fix through science, but several upcoming projects might have some web involvement; so I thought I should brush up. The following are some reflections on that experience.

The task I set myself was to make a micro CMS (it is currently named MojoCMS, but I’m not sure I like that), leaving most of the heavy lifting to freely available Javascript libraries. I didn’t think I would be especially good at writing the actual interface, but rather the routing and functionality would be my task. In a strange way, the result was a kind of nostalgic Perl experience; Perl was the glue in my project again, not the main/only language involved.

I used several great libraries, jQuery of course, jQuery-UI for a small part, HumaneJS for notifications (works great for websocket responses!) and PageDown for a real-time markdown renderer. FYI, PageDown is the editor from the StackOverflow team. These projects make life much easier, I can’t imagine writing that kind of Javascript by hand!

I must say, Javascript still eludes me. I can parrot it, but I’m sure I’m not doing it correctly. I think the problem lies with its dependence on the HTML/browser that is running it; the odd way that the language doesn’t have a use command, and that “page”-globals can be used, still feels odd. I can definitely see the need for jQuery, but that adds even further cognitive dissonance. Anyway, I think most of this is my shortcoming, not its.

HTML5/CSS3 on the other hand is brilliant. Its easy to make the markup do what you mean without too many machinations. Of course I pull in some libraries for that too, namely Bootstrap.

Back to the Perl of it though, I must say I have high marks for Mojolicious, for many reasons, but the highest are for Websockets! Now I know Mojolicious didn’t invent them, but it makes them easy. Using Websockets I was able to make the “save page” and “update main nav” windows save without reloading. That was rather cute and feels modern.

The biggest point I want to make (long ramblings aside) is my most recent addition: DBIx::Class. I’m a scientist, not a database admin. I have setup some PHP CMSes and have used mysql just enough to get them started; terrified the entire time. So much so that I started my CMS project with the idea of using DBM::Deep for as long as possible. Soon enough though, I was nesting hash-keys three deep and wishing I had objects; if I hadn’t needed persistence I would have reached for Moose long before.

I investigated KiokuDB, and while I had some hope for it, I think I would need someone sitting in on the setup process with me. Then I remembered, throughout YAPC::NA along with the list of my favorite Modern Perl modules, everyone else adds DBIx::Class. Ok they can’t all be wrong. They weren’t. Sure the syntax is a little different from Moose, but its not that hard. The payoff for me started even before running the site, in the deploy command. With a simple script I can create all the necessary tables and inject sample content without ever needing to write SQL! After this the ORM quickly and easily replaced the DBM::Deep vestiges throughout the code, and just like that I have readable, OO structures for users, pages and the menu configuration.

Anyway, if anyone wants to play with MojoCMS (or suggest a better name!) feel free. It is still very rough but I may try to see it forward a little further. Passwords are stored in the clear for now, so be careful! But this is my next task. After that and some other work on users (like being able add them through the website!) the thing might even be able to host a small site. Not bad for a one-week side project!

About Joel Berger

user-pic As I delve into the deeper Perl magic I like to share what I can.