Running a non-blocking command in Mojolicious

Mojolicious is an awesome framework designed from the ground up to be non-blocking. During web development it is sometimes desirable to run a command and do something with the results. However, with methods such as "system" and "open" the command will block concurrent requests.

A non-blocking process execution is possible with background processes; however, the process must be managed and if the UserAgent needs updating, then long-polling will need to be utilized.

Below is a complete Mojolicious::Lite program that does this. In words, the app generates a utility script on-the-fly that a) sleeps; b) writes to a state file; and c) sleeps at the the end. This means that our state file will be around while the program is running. This mimics several use-cases, for example creating a thumbnail.

The state file can be inspected using events provided by Mojo::IOLoop::ProcBackground. The alive event is a heartbeat for the process and the dead event happens when the process stops.


my $proc = $self->stash->{_proc} = Mojo::IOLoop::ProcBackground->new;

# Every so often we get a heartbeat from the background process
$proc->on(alive => sub {
    my ($proc) = @_;

    # ...
});

# When the process terminates, we get this event
$proc->on(dead => sub {
    my ($proc) = @_;

    # ...

    $self->finish;   # The GET request is done now
});

# Start our process
$proc->run([$^X, $script, $statefile]);
Here is the full example:
C:\>perl poll.pl
[Thu Feb 13 22:37:51 2014] [info] Listening at "http://*:5555".
Server available at http://127.0.0.1:5555.
[Thu Feb 13 22:37:55 2014] [debug] GET "/run".
[Thu Feb 13 22:37:55 2014] [debug] Routing to a callback.
[Thu Feb 13 22:37:55 2014] [debug] 200 OK (0.001246s, 802.568/s).
[Thu Feb 13 22:38:08 2014] [debug] Done: C:\DOCUME~1\ADMINI~1\LOCALS~1\Temp\done.3676
[Thu Feb 13 22:38:08 2014] [debug] Finished

...

C:\Documents and Settings\Administrator>mojo get "http://127.0.0.1:5555/run"
Starting...
Done C:\Documents and Settings\Administrator>

The code was tested in OS X and Win XP.


use Mojolicious::Lite;

use Mojo::IOLoop::ProcBackground;

use File::Temp;
use File::Spec;
use Proc::Background;

any '/run' => sub {
        my $self = shift;

        # Setup our request to take a while
        Mojo::IOLoop->stream($self->tx->connection)->timeout(30);
        $self->render_later;

        $self->on(finish => sub {
            $self->app->log->debug("Finished");
        });

        # We want the UserAgent to see something as soon as possible
        $self->res->code(200);
        $self->res->headers->content_type('text/html');
        $self->write_chunk("Starting...
\n"); # This is our utility script that will run in the background my $tmp = File::Temp->new(UNLINK => 0, SUFFIX => '.pl'); my $statefile = $self->stash->{_statefile} = File::Spec->catfile(File::Spec->tmpdir, "done"); print($tmp 'sleep(10); $f="$ARGV[0].$$"; open($fh, ">", $f); sleep(3)'); my $script = $tmp->filename; undef($tmp); # Thanks CPAN.. :) The magic happens in Proc::Background my $proc = $self->stash->{_proc} = Mojo::IOLoop::ProcBackground->new; # Every so often we get a heartbeat from the background process $proc->on(alive => sub { my ($proc) = @_; my $pid = $proc->proc->pid; my $statefile = $self->stash->{_statefile} . ".$pid"; if (-f $statefile) { $self->write_chunk("Done"); $proc->unsubscribe("alive"); } }); # When the process terminates, we get this event $proc->on(dead => sub { my ($proc) = @_; my $pid = $proc->proc->pid; my $statefile = $self->stash->{_statefile} . ".$pid"; $self->app->log->debug("Done: $statefile"); $self->finish; }); # Start our process $proc->run([$^X, $script, $statefile]); }; # Run the app push(@ARGV, 'daemon', '-l', 'http://*:5555') unless @ARGV; app->log->level("debug"); app->secrets(["I Knos# you!!"]); app->start;

It should be noted that there is more than one way to run a process. For example, Mojo::IOLoop::ReadWriteFork can be used to follow the process STDOUT/STDERR; and Mojo::IOLoop::ProcBackground simply runs a command in the background and allows for job control; and Mojo::IOLoop::ForkCall can be used to execute arbitrary perl in an async fashion (you can run a command, as well) and have programmatic control over the output.

In summary, one allows for STDOUT/STDERR events; one simply runs a command; and another allows for arbitrary perl to be utilized.

Fetch me a cloud

At times, it is desirable to access any file on your box. Forgot to put it on dropbox? Then what do you do? Enter FileBeagle. It is your own personal cloud and written in Perl and Mojolicious. The install is copying one file onto your box and running the executable. The only other file that is permanent is the database that is created.

There is currently a Lite edition with a Pro version coming soon. The Lite version is free.

Announce CloseBargains.com

It has been said that we need more Perl startups. I agree and have written CloseBargains.com - what better way to get a heapin helpin of local coupons than Perl, Mojolicious, Linux, and some 8coupons API goodness served up by hypnotoad. The frontend is done in jQuery Mobile; another buzzword that needs adding is that Backbone.js needs to be integrated for coolness factor.

Headless Selenium testing with PhantomJS

As you know, Selenium is a marvelous library for automating a browser. It can be combined with Test::More and PhantomJS to provide a headless test suite.

An example script looks like:


#!/opt/perl

use Selenium::Remote::Driver;
use Selenium::Remote::WDKeys;

use Test::More;
use MIME::Base64;

my $driver = Selenium::Remote::Driver->new();

$driver->set_implicit_wait_timeout(3000);

$driver->get('http://www.google.com');
like($driver->get_title(), qr/^Google$/, "Arrived at Google homepage");

my $elem = $driver->find_element("input[name=q]", "css");
$elem->send_keys("Mojolicious");

open(my $fh,'>','mojoText.png');
binmode($fh);
my $png_base64 = $driver->screenshot();
print($fh MIME::Base64::decode_base64($png_base64));
close($fh);

$elem->send_keys(KEYS->{'enter'});

$elem = $driver->find_element("#resultStats", "css");
like($elem->get_text(), qr/About.*results/, "Got some results");

open($fh,'>','mojoResults.png');
binmode($fh);
my $png_base64 = $driver->screenshot();
print($fh MIME::Base64::decode_base64($png_base64));
close($fh);

$driver->quit();

done_testing();

This works under the assumption that PhantomJS is running like so:

[bpm@s001] c:~>./phantomjs --webdriver=4444
PhantomJS is launching GhostDriver...
Ghost Driver running on port 4444

The output looks like:

[bpm@s001] c:~/playground>/opt/perl google.pl 
ok 1 - Arrived at Google homepage
ok 2 - Got some results
1..2
[bpm@s001] c:~/playground>

Selenium WebDriver with automated Basic Auth credentials

Selenium is a marvelous library for automating browsers. Perl has an interface via Selenium::Remote::Driver. A short script looks like:


use Selenium::Remote::Driver;

my $driver = new Selenium::Remote::Driver;
$driver->get(‘http://www.google.com’);
print $driver->get_title();

$driver->quit();

Under the assumption that a selenium server is found, this will launch google and print the title. It works very well for a lot of scenarios. One scenario where it has trouble is with Basic Auth and credentials. The browser usually outputs a popup requiring user-input.

This user-input can be automated via OS specific libraries. Under Windows and ActiveState the PerlScript (one word) interface can be utilized to realize a credential entry system.

The code below will launch a browser and then input the credentials for http://test.webdav.org/auth-basic. It has been tested on Windows 7, Firefox, Selenium WebDriver 2.0.25, and ActiveState 5.14.2.


use Selenium::Remote::Driver;
use Win32::Process;
use Win32;

$WshArgs = $WScript->{Arguments};
my $password = $WshArgs->Item(0);

if ($password) {
    $WshShell = $WScript->CreateObject(“WScript.Shell”);

    while (1) {
        if ($WshShell->AppActivate(“Authentication Required”)) {
            $WshShell->SendKeys(“user1”);
            $WshShell->SendKeys(“{TAB}”);
            $WScript->Sleep(100);

            $WshShell->SendKeys(“user1”);
            $WScript->Sleep(100);

            $WshShell->SendKeys(“{ENTER}”);

            exit;
        }

        select(undef, undef, undef, 0.3);
    }

    exit;
}

sub ErrorReport{
    $WScript->Echo(Win32::FormatMessage( Win32::GetLastError() ));
}

my $driver = Selenium::Remote::Driver->new();

my $ProcessObj;
Win32::Process::Create($ProcessObj,
                “c:\windows\system32\cmd.exe”,
                “start cmd /k cscript keystrokes.pls -password”,
                0,
                NORMAL_PRIORITY_CLASS | DETACHED_PROCESS,
                “.”) || die ErrorReport();

$driver->get(‘http://test.webdav.org/auth-basic’);

$ProcessObj->Kill(0);

$WScript->Echo($driver->get_title());
$driver->quit();