Dylan Cali

A few months ago I released ZMQ::FFI, zeromq bindings for Perl that use FFI::Raw instead of XS. There's been some interest, so I thought I would write a short post to announce the module beyond the zeromq mailing list.

In another post I'll discuss FFI::Raw itself, what it currently provides, as well as some advanced use cases that came up during the development of ZMQ::FFI. If reverse depends on metacpan is anything to go by, it seems many people aren't aware of this module. In fact, at the time of this writing ZMQ::FFI is the only module using it:

That's a shame, as it's a terrific module which finally makes FFI in Perl feasible. It effectively eliminates the need for XS in many cases, and it's sufficiently baked to be considered "production ready."

ZMQ::FFI

This post assumes existing knowledge of zeromq. If you have no idea what zeromq is, or what it's used for, you can get started here and here.

The description for ZMQ::FFI is already pretty concise, so let's start with that:

ZMQ::FFI exposes a high level, transparent, OO interface to zeromq independent of the underlying libzmq version. Where semantics differ, it will dispatch to the appropriate backend for you. As it uses ffi, there is no dependency on XS or compilation.

What this means in practice is that it doesn't matter if you have zeromq 2/3/4 installed; you can use the same consistent interface. To demonstrate this, as well as ZMQ::FFI's basic usage, let's start with a simple send/recv example:

sendrecv1.pl

use ZMQ::FFI;
use ZMQ::FFI::Constants qw(ZMQ_REQ ZMQ_REP);

my $endpoint = "ipc://zmq-ffi-$$";
my $ctx      = ZMQ::FFI->new();
my $version  = join '.', $ctx->version;

my $req = $ctx->socket(ZMQ_REQ);
$req->connect($endpoint);

my $rep = $ctx->socket(ZMQ_REP);
$rep->bind($endpoint);

$req->send($version);
say $rep->recv();

Pretty basic: we setup a simple request/reply connection, send the zeromq version string as the message, receive it, and print it. $ctx->version() returns a list of ($major, $minor, $patch) and hence the join. On my Arch Linux box with zeromq 4.0.3 installed I get:

$ perl sendrecv1.pl
4.0.3

As expected.

Now what if an older libzmq version is installed? Say 2.x which has an incompatible ABI. I can simulate this by setting my LD_LIBRARY_PATH to the location of the older .so:

$ ls ~/git/zeromq2-x/src/.libs/libzmq.so*
/home/calid/git/zeromq2-x/src/.libs/libzmq.so
/home/calid/git/zeromq2-x/src/.libs/libzmq.so.1
/home/calid/git/zeromq2-x/src/.libs/libzmq.so.1.0.1
$ export LD_LIBRARY_PATH=~/git/zeromq2-x/src/.libs/
$ perl sendrecv1.pl
2.2.1

Success. I was able to reuse the same code and ZMQ::FFI was able to gracefully handle the ABI differences between versions.

It's important to point out that contexts are isolated from one another, ensuring you can have several in the same process. In fact, it's even possible to create contexts using different libzmq versions:

sendrecv2.pl

use ZMQ::FFI;
use ZMQ::FFI::Constants qw(ZMQ_REQ ZMQ_REP);

my $endpoint = "ipc://zmq-ffi-$$";

# send using 2.x context
my $req_ctx     = ZMQ::FFI->new( soname => 'libzmq.so.1' );
my $req_version = join '.', $req_ctx->version;

my $req = $req_ctx->socket(ZMQ_REQ);
$req->connect($endpoint);
$req->send($req_version);

# receive using 4.x context
my $rep_ctx     = ZMQ::FFI->new( soname => 'libzmq.so.3' );
my $rep_version = join '.', $rep_ctx->version;

my $rep = $rep_ctx->socket(ZMQ_REP);
$rep->bind($endpoint);

say join " ",
    $rep_version, "context",
    "received message from",
    $rep->recv(), "context";

$ perl sendrecv2.pl
4.0.3 context received message from 2.2.1 context

Using the soname attribute I've explicitly indicated the libzmq.so I want to use to create each context. If the .so doesn't already exist on your ld path manually adding it via LD_LIBRARY_PATH is one option, but you can also reference it directly using an absolute path. So in the example above if I change

ZMQ::FFI->new( soname => 'libzmq.so.3' )

to

ZMQ::FFI->new( soname => '/home/calid/git/zeromq3-x/src/.libs/libzmq.so' )

my output changes to:

$ perl sendrecv2.pl
3.2.5 context received message from 2.2.1 context

Being able to set the soname is especially useful when you have parallel zeromq installations, all on your ld path. The first version found may not be the one you actually want to use. By explicitly setting the soname you can eliminate any ambiguity.

This extends to zeromq 3 vs zeromq 4 which have the same major soname (libzmq.so.3). The solution here is to specify the complete soname, major.minor.patch:

my $req_ctx = ZMQ::FFI->new( soname => 'libzmq.so.3.0.0' );
...
my $rep_ctx = ZMQ::FFI->new( soname => 'libzmq.so.3.1.0' );

$ perl sendrecv2.pl
4.0.3 context received message from 3.2.5 context

The identical soname is a result of zeromq 4 having an ABI that is backward compatible with zeromq 3.

nonblocking

The above examples show how we can flexibly deal with different libzmq versions. The zmq code itself is pretty trivial though. Let's do something a little more sophisticated and exercise the asynchronous nature of zeromq. ZMQ::FFI provides convenience functions that make nonblocking semantics natural. To demonstrate this we can integrate ZMQ::FFI with AnyEvent:

nonblock1.pl

use ZMQ::FFI;
use ZMQ::FFI::Constants qw(ZMQ_PUSH ZMQ_PULL);

use AnyEvent;
use EV;

my $endpoint = "ipc://zmq-ffi-$$";
my $ctx      = ZMQ::FFI->new();

my $push = $ctx->socket(ZMQ_PUSH);
$push->connect($endpoint);

my $pull = $ctx->socket(ZMQ_PULL);
$pull->bind($endpoint);

my $recvd = 0;
my $w = AE::io $pull->get_fd, 0, sub {
    while ($pull->has_pollin) {
        print $pull->recv();

        $recvd++;
        if ( $recvd == 3 ) {
            EV::unloop;
            print "\n";
        }
        else {
            print '.';
        }
    }
};

my $t = AE::timer 0, 0, sub {
    $push->send($_) for $push->version;
};

EV::run;

$ perl nonblock1.pl
4.0.3

We're still sending the zeromq version as the messsage, but how we're sending it is very different:

• Instead of request/reply we're using push/pull
• The version parts are now being sent one at a time and joined on the receiving end
• Rather than directly sending/receiving, we're doing these through an event loop (sending on a timer, polling and then receiving only when there's data)

Still, how we integrated with AnyEvent is the most important part:

• When setting up our io watcher we used the pull socket's get_fd method, which returns the underlying zmq file descriptor to poll on
• In the watcher's callback we used the has_pollin method to guarantee we receive all the messages currently available on the socket

The second point is critical: By continuing to receive until has_pollin returns false, we ensure all outstanding zmq events on the socket have been exhausted. If we return control to the event loop without doing this the edge-triggered state of the zmq fd may not get reset. The consequence of leaving the fd in an intermediate state is that it will not show as readable again, and the io callback will never fire.

After all of that let's quickly prove we're still version agnostic:

$ LD_LIBRARY_PATH=~/git/zeromq2-x/src/.libs/ perl nonblock1.pl
2.2.1

Still ok!

Final points

You may have noticed I never explicitly called close or destroy. This is because cleanup is handled automatically in the Moose DEMOLISH methods. The close/destroy methods do exist though, if for some reason you can't wait for the objects to get reaped.

Likewise, I did no error checking. Error codes are checked for you, and if an error does occur ZMQ::FFI will raise an appropriate exception.

That wraps up this whirlwind tour of ZMQ::FFI. Do let me know any feedback on the post or the module. I did not demo multipart messages or pub/sub, but you can find examples of these as well as more detailed documentation here:

• https://github.com/calid/zmq-ffi
• https://metacpan.org/pod/ZMQ::FFI

My next post will delve into FFI::Raw and how I was able to use it to make ZMQ::FFI possible.

Cheers!