Preallocating scalars

I'm using the fabulous FFI::Platypus to interface to a C routine which uses caller-allocated buffers to return data. While FFI::Platypus transparently translates Perl arrays to C arrays and back, the buffers are used only to return data, so I only need the C-to-Perl conversion and not the Perl-to-C conversion.

The first step is to efficiently allocate a buffer of a given size in Perl (the last step, converting the retuned data in the buffer to Perl, is done straightforwardly with unpack).

If you do your due diligence, you'll find a link to an old PerlMonks post, which provides the following recipe:

my $str;
vec( $str, $length, 8 ) = 0;
$str = '';

That seemed like too many things to type, so I thought back to my telemetry unpacking days and of course, pack to the rescue. There's at least a couple of ways to do it with pack.

This one writes $length null bytes:

my $str = pack( "x$length" );

And this one null fills to a given absolute position

my $str = pack( ".", $length );

I tend to like this one, as it doesn't require string interpolation.

Then, there's Convert::String::grow:

my $str;
grow $str, $length;

which is a thin wrapper around Perl's internal sv_grow routine.

So which is the fastest? Here are benchmarks (Perl 5.28.1) for various buffer lengths.

Buffer size = 1000 bytes
                       Rate    vec pack_x pack_dot   grow
vec      3.31756e+06+-160/s     -- -50.2%   -57.4% -66.5%
pack_x   6.6639e+06+-1700/s 100.9%     --   -14.4% -32.7%
pack_dot  7.784e+06+-1800/s 134.6%  16.8%       -- -21.4%
grow     9.8974e+06+-2900/s 198.3%  48.5%    27.2%     --


Buffer size = 10000 bytes
                        Rate    vec pack_x pack_dot   grow
vec         2.82529e+06+-0/s     -- -10.0%   -12.9% -72.1%
pack_x     3.139e+06+-1200/s  11.1%     --    -3.3% -69.0%
pack_dot  3.24488e+06+-610/s  14.9%   3.4%       -- -67.9%
grow     1.01173e+07+-2200/s 258.1% 222.3%   211.8%     --


Buffer size = 100000 bytes
                       Rate pack_dot  pack_x     vec   grow
pack_dot       213512+-94/s       --   -1.5%   -3.3% -97.9%
pack_x         216718+-87/s     1.5%      --   -1.8% -97.9%
vec           220760+-180/s     3.4%    1.9%      -- -97.8%
grow     1.01299e+07+-740/s  4644.4% 4574.2% 4488.6%     --


Buffer size = 1000000 bytes
                        Rate      vec pack_dot   pack_x   grow
vec           20776.6+-3.2/s       --    -0.2%    -0.3% -99.8%
pack_dot     20822.7+-0.75/s     0.2%       --    -0.1% -99.8%
pack_x         20843+-0.73/s     0.3%     0.1%       -- -99.8%
grow     1.01142e+07+-2200/s 48580.7% 48473.0% 48425.7%     --


Buffer size = 10000000 bytes
                       Rate  pack_dot    pack_x       vec    grow
pack_dot    1886.17+-0.24/s        --     -0.1%     -6.7% -100.0%
pack_x      1887.12+-0.21/s      0.1%        --     -6.7% -100.0%
vec         2022.37+-0.48/s      7.2%      7.2%        -- -100.0%
grow     1.01198e+07+-750/s 536423.5% 536153.1% 500289.6%      --

Not surprisingly, Convert::String::grow blows everything else out of the water.

If you need a core Perl solution, pack( '.', ...) is faster for smaller buffers and comparable to vec for larger ones, so provides the best overall solution. Unfortunately both solutions both allocate and clear the memory. I can't think of a core Perl solution which will just allocate the memory.

And here's the code:

use Benchmark::Dumb qw[ cmpthese ];

use strict;
use warnings;

use Convert::Scalar qw[ grow ];

for my $length ( 1e3, 1e4, 1e5, 1e6, 1e7 ) {

    print "\n\nBuffer size = $length bytes\n";

    cmpthese(
        1000.001,
        {
            pack_x => sub {
                my $str = pack( "x$length" );
            },
            pack_dot => sub {
                my $str = pack( ".", $length );
            },
            vec => sub {
                my $str;
                vec( $str, $length, 8 ) = 0;
                $str = '';
            },
            grow => sub {
                my $str;
                grow $str, $length;
            }
        } );
}

print "\n";

How fast can you try?

I just saw the release of Aristotle's Try::Tiny::Tiny to CPAN, which aims to speed up Try::Tiny. That led me to wonder how fast the various Try* modules were. I cannibalized the benchmark code from Try::Catch, and off I went.

Extracting values from a list of (key, value) pairs

If you need ordered key, value pairs, you can either use something like Tie::IxHash or a simple array of key, value pairs. I found myself in the situation where I needed to extract just the keys from such an array.

There are a number of ways to do it, but which is the fastest? I tried a few pure Perl approaches, as well as List::Util::pairkeys (which as of this writing isn't yet in a stable release of Perl, assuming that List::Util remains in the core). The pure Perl approaches either use various means of flipping a binary toggle, or splice()ing through a sacrificial copy of the array.

Here we go:

#!/usr/bin/env perl

use 5.10.1;

use strict;
use warnings;

use List::Util qw[ pairkeys ];

use Benchmark qw[ timethese cmpthese ];

my $N = 10000;

my @N = 1 .. $N;

cmpthese(
    timethese(
        10000,
        {

            'splce' => sub {
                my @v;
                my ( $k, $v );
                my @N = @N;
                push @v, $k while ( ( $k, $v ) = splice( @N, 0, 2 ) );

            },

            '%m' => sub {
                use integer;
                my $flip = 0;
                my @v    = map { ++$flip % 2 ? $_ : () } @N;
            },

            '%g' => sub {
                use integer;
                my $flip = 0;
                my @v    = grep { ++$flip % 2 } @N;
            },

            '1-m' => sub {
                use integer;
                my $flip = 0;
                my @v
                  = map { ( $flip = 1 - $flip ) ? $_ : () } @N;
            },

            '1-g' => sub {
                use integer;
                my $flip = 0;
                my @v    = grep { $flip = 1 - $flip } @N;
            },

            'ff' => sub {
                use integer;
                my $flip = 0;
                my @v    = grep { $flip = !( $flip .. $flip ) } @N;
            },

            'xor' => sub {
                use integer;
                my $flip = 0;
                my @v    = grep { $flip ^= 1 } @N;
            },

            'for' => sub {

                use integer;
                my @v;
                for( my $i = 0 ; $i < @N ; $i+=2 ) {
                    push @v, $N[$i];
                }
            },

            'idxp' => sub {

                use integer;
                my @v;
                push @v, $N[2 * $_] for 0..(@N/2)-1;
            },

            'idxm' => sub {

                use integer;
                my @v;
                @v = map { $N[2 * $_] } 0..(@N/2)-1;
            },

            'idxa' => sub {
                use integer;
                my @v;
                $v[$_] = $N[2 * $_] for 0..(@N/2)-1;
            },

            'pkey' => sub {
                my @v = pairkeys @N;
            },

        } ) );

And here are the results:

        Rate splce   ff   %m  1-m idxm   %g  for  xor idxa  1-g idxp pkey
splce  497/s    --  -3%  -5% -27% -30% -33% -40% -52% -53% -54% -59% -74%
ff     514/s    3%   --  -2% -25% -28% -30% -38% -50% -52% -53% -58% -74%
%m     524/s    6%   2%   -- -23% -26% -29% -37% -49% -51% -52% -57% -73%
1-m    681/s   37%  33%  30%   --  -4%  -8% -18% -34% -36% -37% -44% -65%
idxm   712/s   43%  39%  36%   5%   --  -3% -14% -31% -33% -34% -41% -63%
%g     738/s   49%  44%  41%   8%   4%   -- -11% -29% -31% -32% -39% -62%
for    831/s   67%  62%  58%  22%  17%  13%   -- -20% -22% -24% -32% -57%
xor   1037/s  109% 102%  98%  52%  46%  41%  25%   --  -3%  -4% -15% -47%
idxa  1067/s  115% 108% 104%  57%  50%  45%  28%   3%   --  -2% -12% -45%
1-g   1086/s  119% 111% 107%  59%  52%  47%  31%   5%   2%   -- -11% -44%
idxp  1214/s  144% 136% 131%  78%  70%  64%  46%  17%  14%  12%   -- -37%
pkey  1942/s  291% 278% 270% 185% 173% 163% 134%  87%  82%  79%  60%   --
  • If you've got List::Util::pairkeys, use it.
  • Array indexing is surprisingly fast.
  • Pushing onto the end of an array is significantly faster than direct assignment into it.
  • grep is faster than map; not too surprising
  • Remainders (%) are slower then subtraction; not too surprising.
  • Binary exclusive-or is comparable to subtraction, which is a bit surprising
  • In this application use integer provided a significant boost in speed.
  • The flipflop operator is surprisingly slow
  • splice() and flipflop are essentially tied. splice() is slowed down significantly by needing to make a copy of the array. (If I added the copy cost to the others it moved up two slots in the rankings). It would be even slower if the array (or its elements) were larger. When Copy on Write (COW) makes it into Perl, that difference should diminish.

Automating POD previewing

I find it difficult to review documentation written in a markup language. I need to see that final pretty rendition of it as a manual page, PDF file or HTML web page before I can get a feeling if the thing makes sense. I also like to see that final rendition while I'm writing the document, and switching between the editor and the command line to render the documentation is painful.

Luckily most modern OS's provide a means of monitoring filesystems for changes. I've put together a rather simple bash script which uses Linux's inotify system and App::Pod2CpanHtml to generate HTML output whenever I save a .pod or .pm file. I use Firefox with the Auto Reload plugin to view the files. For grins the script also has an option to pop up a desktop notification when things change, so I know something has happened.

Perl and Perl Module Administration in the Modern Era

I recently gave a talk on perlbrew, cpanminus, and local::lib at $work. I've uploaded the talk to Speaker Deck