berrybrew, the Perlbrew for Windows has been updated

berrybrew, the Perlbrew for Windows has been updated.

The significant new feature is the ability to automatically fetch the available Perl instances using Strawberry's new JSON releases file. This does not happen every time you use available, as I didn't want to force a user to have to be connected to the Internet while using berrybrew. Instead, I added a new berrybrew fetch command that does the work.

Things to know:

  • we now only list the most recent point version of each major release (this may change in the future)
  • we list the 64bit, 32bit and the PDL version of each major release, where available (this may also change to include other versions, such as USE_LONG_DOUBLE etc)
  • upon the first fetch, if any of the currently installed instances are lower than the most recent point release per version, we register them as custom installs. This allows them to continue to be maintained by berrybrew, and don't fall off the map as orphaned instances

I have also introduced changes to berrybrew upgrade. Previously, we'd make a backup of all JSON configuration files, then after the binary upgrade, we'd put those files back into the data/ production directory. Now, we still back up all of the configs, but the only one restored is the perls_custom.json file. If any custom modifications were made to any of the other config files, they will have to be manually merged back into the equivalent file in the data/ directory from the backups. That said, it is my understanding that very few people modify the configs anyway, so this shouldn't concern many.

To install without already having berrybrew installed, either git clone https://github.com/stevieb9/berrybrew or download the pre-built zip archive, and unzip it to anywhere other than c:\berrybrew, then:

  • cd repo_dir
  • bin\berrybrew.exe config
  • close cmd window, open new one
  • berrybrew install ...
  • close cmd, open new one
  • and you're good to go

Note that because I maintain David Farrell's original repository of the project that mine was based off of, it has been updated as well.

Here's the process to do an upgrade from a previous version. First show the list that appears within the current version:

 > berrybrew available

The following Strawberry Perls are available:

        5.24.1_64       [installed] *
        5.24.1_32
        5.24.0_64       [installed]
        5.24.0_64_PDL
        5.24.0_32
        5.22.3_64
        5.22.3_32
        5.22.2_64
        5.22.2_64_PDL
        5.22.2_32
        5.22.1_64       [installed]
        5.22.1_32
        5.20.3_64
        5.20.3_64_PDL
        5.20.3_32
        5.20.3_32_PDL
        5.18.4_64
        5.18.4_32
        5.16.3_64
        5.16.3_32
        5.14.4_64
        5.14.4_32
        5.12.3_32
        5.10.1_32

* Currently using

Then, upgrade:

> cd berrybrew_repo

Either use git pull, or extract out the zip file over the existing installation. The older version of upgrade will trample on the new configuration file, so if you use berrybrew upgrade, download the config.json file from the repository, and place it into your data/ directory.

berrybrew available will show if you have orphans that aren't listed in the new available file:

WARNING! orphaned perl installations:


  5.22.1_64
  5.24.0_64

Run the fetch, and your previous installs that don't line up with the current Perls are registered for use with berrybrew.

> berrybrew fetch

> berrybrew available

The following Strawberry Perls are available:

        5.24.1_64       [installed] *
        5.24.1_64_PDL
        5.24.1_32
        5.22.3_64
        5.22.3_64_PDL
        5.22.3_32
        5.20.3_64
        5.20.3_64_PDL
        5.20.3_32
        5.18.4_64
        5.18.4_32
        5.16.3_64
        5.16.3_32
        5.14.4_64
        5.14.4_32
        5.12.3_32
        5.10.1_32
        5.8.9_32
        5.22.1_64       [custom] [installed]
        5.24.0_64       [custom] [installed]

* Currently using

You can now...

berrybrew switch $version

... to them, and continue using them as normal.

Wrapping a C shared library with Perl and XS

This tutorial shows how to wrap a C shared library using XS and Perl (including creating a trivial test shared library).

The first 3/4 is the actual wrapping itself. The second part includes the C code and build commands to create the trivial shared library used in this tutorial.

Relatively, I am still very new to all of this, as it's a pretty complex world. Before I started, I didn't have any real C experience, so I've been dealing with that learning curve at the same time, so I know there are better and more efficient ways of doing what I do, and would appreciate any feedback.

I'll get right to it. Here's an overview:

  • find something to wrap. In this case, I've written a shared C library called xswrap (this code/info appears later in the tutorial)
  • create a shell distribution that'll allow us to load our eventual XS code, which in turn has wrapped the C library
  • update relevant files to make things hang together
  • run into a function that can't be returned to Perl as-is, so we learn how to write our own C/XS wrapper so we can get what we need
  • package it all together into a distribution

The actual C code is irrelevant at this point, but knowing the definitions in use is, so here they are for the xswrap library:

int mult (int x, int y);
void speak (const char* str);
unsigned char* arr (); // returns (0, 1, 2)

Create a new shell distribution

I use Module::Starter:

module-starter \
    --module=XS::Wrap \
    --author="Steve Bertrand" \
    --email=steveb@cpan.org \
    --license=perl

Now change into the new XS-Wrap directory, which is the root directory of the new dist. The Perl module file is located at lib/XS/Wrap.pm. I've removed a bunch of stuff for brevity, but the shell looks something like this:

package XS::Wrap;

use warnings;
use strict;

our $VERSION = '0.01';

Create the base XS file

I use Inline::C to do this for me, as like most Perl hackers, I'm often lazy. Note the flags in use. clean_after_build tells Inline to not clean up the build directory (_Inline after build). This allows us to fetch our new .xs file. name is the name of the module we're creating this XS file for.

use warnings;
use strict;

use Inline Config =>
           disable => clean_after_build =>
           name => 'XS::Wrap';
use Inline 'C';

__END__
__C__

#include <stdio.h>
#include <xswrap.h>

The resulting XS file is located in _Inline/build/XS/Wrap/Wrap.xs. Copy it to the root directory of the dist:

cp _Inline/build/XS/Wrap/Wrap.xs .

Here's what our base XS file looks like. It doesn't do anything yet, but we'll get there:

#include "EXTERN.h"
#include "perl.h"
#include "XSUB.h"
#include "INLINE.h"

#include <stdio.h>
#include <xswrap.h>

MODULE = XS::Wrap  PACKAGE = main

PROTOTYPES: DISABLE

See the PACKAGE = main there? Change main to the name of our dist, XS::Wrap.

Adding the functions from the shared library to XS

Now we need to define our C functions within the XS file. After I've done that using the C definitions for the functions above, my XS file now looks like this:

#include "EXTERN.h"
#include "perl.h"
#include "XSUB.h"
#include "INLINE.h"

#include <stdio.h>

MODULE = XS::Wrap  PACKAGE = XS::Wrap

PROTOTYPES: DISABLE

int mult (x, y)
    int x
    int y

void speak (str)
    const char* str

unsigned char* arr ()

Note that at this point, because we're not using Inline anymore, you can remove the include for the INLINE.h header file. However, in our case, we're going to be using some Inline functionality a bit later on, so instead of removing that, copy the INLINE.h file to the same directory we copied our XS file into: cp _Inline/build/XS/Wrap/INLINE.h .

Readying the module file for use

Now we have some work to do to pull in the XS, wrap the functions, and export them. Note that you do not need to wrap the functions with Perl, you can export them directly as depicted in the XS file if you wish, as long as you know you don't need to add any further validation or functionality before the XS imported C function is called. I'll wrap all three. The functions that each wrapped function calls is the literal C function, as advertised through the XS file we hacked above.

use warnings;
use strict;

our $VERSION = '0.01';

require XSLoader;
XSLoader::load('XS::Wrap', $VERSION);

use Exporter qw(import);

our @EXPORT_OK = qw(
    my_mult
    my_speak
    my_arr
);

our %EXPORT_TAGS;
$EXPORT_TAGS{all} = [@EXPORT_OK];

sub my_mult {
    my ($x, $y) = @_;
    return mult($x, $y);
}
sub my_speak {
    my ($str) = @_;
    speak($str);
}
sub my_arr {
    my @array = arr();
    return @array;
}

Telling the Makefile to load the external C library

Because we're using an external shared library, we need to add a directive to the Makefile.PL file. Put the following line anywhere in the Makefile.PL's `WriteMakefile() routine:

LIBS => ['-lxswrap'],

Building, installing and initial test

Let's build, install and write a test script for our new distribution.

perl Makefile.PL
make
make install

At this point, if everything works as expected, you're pretty well done. However, in the case here, we're going to unexpectedly run into some issues, and we'll need to do other things before we finalize our distribution.

Test script (example.pl). Very basic, it just tests all three wrapped functions:

use warnings;
use strict;
use feature 'say';

use XS::Wrap qw(:all);

say my_mult(5, 5);

my_speak("hello, world!\n");

my @arr = my_arr();

say $_ for @arr;

Output:

25
hello, world!

Hmmm, something is not right. The arr() C function was supposed to return an array of three elements, 0, 1, 2, but we get no output.

This is because arr() returns an unsigned char* which we can't handle correctly/directly in Perl.

In this case, I will just wrap the arr() function with a new C function (I've called it simply _arr()) that returns a real Perl array based on the output from the original C arr() function. Technically, I won't be returning anything, I'm going to just use functionality from Inline to put the list onto the stack, where Perl automatically picks it up.

To do this, I'll be leveraging Inline again, but with a couple of changes. We change the name, and add also bring in our shared library because we need it directly now.

Returning a Perl array from a C function

use warnings;
use strict;

use Inline config =>
           disable => clean_after_build =>
           name => 'Test';
use Inline ('C' => 'DATA', libs => '-lxswrap');

print "$_\n" for _arr();

 __END__
 __C__

#include <stdio.h>
#include <xswrap.h>

void _arr (){
    unsigned char* c_array = arr();

    inline_stack_vars;
    inline_stack_reset;

    int i;

    for (i=0; i<3; i++){
        inline_stack_push(sv_2mortal(newSViv(c_array[i])));
    }

    inline_stack_done;
}

After I execute that Perl script, I'm left with a new XS file within the _Inline/build/Test/Test.xs. . It looks like this:

#include "EXTERN.h"
#include "perl.h"
#include "XSUB.h"
#include "INLINE.h"

#include <stdio.h>
#include <xswrap.h>

void _arr (){
    unsigned char* c_array = arr();

    inline_stack_vars;
    inline_stack_reset;

    int i;

    for (i=0; i<3; i++){
        inline_stack_push(sv_2mortal(newSViv(c_array[i])));
    }

    inline_stack_done;
}


MODULE = Test  PACKAGE = main  

PROTOTYPES: DISABLE


void
_arr ()
        PREINIT:
        I32* temp;
        PPCODE:
        temp = PL_markstack_ptr++;
        _arr();
        if (PL_markstack_ptr != temp) {
          /* truly void, because dXSARGS not invoked */
          PL_markstack_ptr = temp;
          XSRETURN_EMPTY; /* return empty stack */
        }
        /* must have used dXSARGS; list context implied */
        return; /* assume stack size is correct */

We only need a couple of pieces of it, so get out your CTRL-V and CTRL-C. Here are the sections (cleaned up a bit for brevity) that we need to copy into our real Wrap.xs file.

The C portion:

void _arr (){
    unsigned char* c_array = arr();

    inline_stack_vars;
    inline_stack_reset;

    int i;

    for (i=0; i<3; i++){
        inline_stack_push(sv_2mortal(newSViv(c_array[i])));
    }

    inline_stack_done;
}

The XS portion:

void
_arr ()
        PREINIT:
        I32* temp;
        PPCODE:
        temp = PL_markstack_ptr++;
        _arr();
        if (PL_markstack_ptr != temp) {
          PL_markstack_ptr = temp;
          XSRETURN_EMPTY;
        }
        return;

The C part goes near the top of the XS file, and the XS part goes in the XS section at the bottom. Here's our full XS file after I've merged in these changes:

Finalized XS file

#include "EXTERN.h"
#include "perl.h"
#include "XSUB.h"
#include "INLINE.h"

#include <stdio.h>
#include <xswrap.h>

void _arr (){
    unsigned char* c_array = arr();

    inline_stack_vars;
    inline_stack_reset;

    int i;

    for (i=0; i<3; i++){
        inline_stack_push(sv_2mortal(newSViv(c_array[i])));
    }

    inline_stack_done;
}

MODULE = XS::Wrap  PACKAGE = XS::Wrap

PROTOTYPES: DISABLE

int mult (x, y)
    int x
    int y

void speak (str)
    const char* str

unsigned char* arr ()

void _arr ()
        PREINIT:
        I32* temp;
        PPCODE:
        temp = PL_markstack_ptr++;
        _arr();
        if (PL_markstack_ptr != temp) {
          PL_markstack_ptr = temp;
          XSRETURN_EMPTY;
        }
        return;

So, in our XS, we have four functions. Three that are imported directly from the C shared lib (mult(), speak() and arr()) and one new one written in C locally that wraps an imported XS function (_arr()).

We need to do a quick update to the wrapper in the module file. Change the call to arr() to _arr() in the .pm file within the my_arr() function:

sub my_arr {
    my @array = _arr();
    return @array;
}

Repeat the build/install steps, then test again:

perl example.pl
25
hello, world!

0
1
2

Cool! Our custom C wrapper for arr() works exactly how we want it to. We're ready for release!

Creating a release of our distribution

It's very trivial to do:

rm -rf _Inline
perl Makefile.PL
make
make test
make manifest
make install
make dist

Of course, you have written all of your POD and unit tests before reaching this point, but I digress :)

Creating and building the shared library

Here is the C code used for the above example, including build steps.

The header file (xswrap.h)

int mult (int x, int y);
void speak (const char* str);
unsigned char* arr ();

The .c file (xswrap.c)

#include <stdio.h>
#include <stdlib.h>

int mult (int x, int y){
    return x * y;
}

void speak (const char* str){
    printf("%s\n", str);
}

unsigned char* arr (){
    unsigned char* list = malloc(sizeof(unsigned char) * 3);

    int i;

    for (i=0; i<3; i++){
        list[i] = i;
    }

    return list;
}

The entry point file (main.c) for testing the lib

#include <stdio.h>
#include "xswrap.h"

int main (){
    int ret = mult(5, 5);
    printf("%d\n", ret);

    speak("hello, world!");

    unsigned char* list = arr(); 

    int i;

    for (i=0; i<3; i++){
        printf("%d\n", list[i]);
    }

    return 0;
}

The build/install script (build.sh)

#!/bin/sh
gcc -c -fPIC xswrap.c
gcc -shared -fPIC -Wl,-soname,libxswrap.so -o libxswrap.so xswrap.o -lc
sudo cp libxswrap.so /usr/lib
sudo cp xswrap.h /usr/local/include

Done!

The library and its header file are both now put into directories in your PATH.

To compile the test program:

gcc -o test main.c -lxswrap

...run it:

./test

You're now ready to wrap the library using Perl and XS.

Raspberry Pi becoming more prevalent?

The last half-year or so, I've been hacking on different Integrated Circuits, various small hardware, learning how to wrap specific C software (while ensuring the code remains within the Perl license) in order to bring the Raspberry Pi toward the realm of reasonable Perl programming.

I have written numerous pieces of software to allow this. The first was WiringPi::API. This is the core wrapper that allows you to use the wiringPi library right from your Perl scripts.

I pushed this further…

Send in a Perl aref to C, get back a Perl array (and using the generated XS)

This is a tutorial as much as it is a request for guidance from experienced XS/C/perlguts folks, as TIMTOWTDI, and in this case, likely, a better way.

This will show you how to pass a Perl array reference (aref) into a C function, convert the aref into a C array, work on it, then push it back onto the stack so the C function returns it as a Perl array.

It'll also show that although we bite off of Inline::C, the XS code it generates can be used in your distribution, even without the end-user needing Inline installed.

First, straight to the code. Comments inline for what's happening (or, at least, what I think is happening... feedback welcomed):

use warnings;
use strict;
use feature 'say';

use Inline 'Noclean';
use Inline 'C';

my $aref = [qw(1 2 3 4 5)];

# overwrite the existing aref to
# minimize memory usage

@$aref = aref_to_array($aref);

say $_ for @$aref;


__END__
__C__

void aref_to_array(SV* aref){

    // check if the param is an array reference...
    // die() if not

    if (! SvROK(aref) || SvTYPE(SvRV(aref)) != SVt_PVAV){
        croak("not an aref\n");
    }

    // convert the array reference into a Perl array

    AV* chars = (AV*)SvRV(aref);

    // allocate for a C array, with the same number of
    // elements the Perl array has

    unsigned char buf[av_len(chars)+1];

    // convert the Perl array to a C array

    int i;

    for (i=0; i<sizeof(buf); i++){
        SV** elem = av_fetch(chars, i, 0);
        buf[i] = (unsigned char)SvNV(*elem);
    }

    // prepare the stack

    inline_stack_vars;
    inline_stack_reset;

    int x;

    for (x=0; x<sizeof(buf); x++){

        // extract elem, do stuff with it, 
        // then push to stack

        char* elem = buf[x];
        elem++;        

        inline_stack_push(sv_2mortal(newSViv(elem)));
    }

    // done!

    inline_stack_done;
}

We now get an _Inline directory created within the current working directory, which has a build/ dir and then a sub directory (or multiple, just look at the one with the most recent timestamp). Peek in there, and you'll see a file with an .xs extention. This is the file you want if you want to include your work into a real Perl distribution. This essentially allows one to utilize my favourite feature of Inline::C, which is to build XS code for us, without having to know any XS (or little XS) at all.

After I run the above example, I get this in the XS file (my comments removed):

#include "EXTERN.h"
#include "perl.h"
#include "XSUB.h"
#include "INLINE.h"

void aref_to_array(SV* aref){

    if (! SvROK(aref) || SvTYPE(SvRV(aref)) != SVt_PVAV){
        croak("not an aref\n");
    }

    AV* chars = (AV*)SvRV(aref);

    unsigned char buf[av_len(chars)+1];

    int i;

    for (i=0; i<sizeof(buf); i++){
        SV** elem = av_fetch(chars, i, 0);
        buf[i] = (unsigned char)SvNV(*elem);
    }

    inline_stack_vars;
    inline_stack_reset;

    int x;

    for (x=0; x<sizeof(buf); x++){

        char* elem = buf[x];
        elem++;        

        inline_stack_push(sv_2mortal(newSViv(elem)));
    }

    inline_stack_done;
}

MODULE = c_and_back_pl_f8ff  PACKAGE = main  

PROTOTYPES: DISABLE


void
aref_to_array (aref)
    SV *    aref
        PREINIT:
        I32* temp;
        PPCODE:
        temp = PL_markstack_ptr++;
        aref_to_array(aref);
        if (PL_markstack_ptr != temp) {
          /* truly void, because dXSARGS not invoked */
          PL_markstack_ptr = temp;
          XSRETURN_EMPTY; /* return empty stack */
        }
        /* must have used dXSARGS; list context implied */
        return; /* assume stack size is correct */

To note is the following line:

MODULE = c_and_back_pl_f8ff  PACKAGE = main

That dictates the name of the module you're creating the XS for. You'll want to change it to something like:

MODULE = My::Module  PACKAGE = My::Module

...then put that file in the root of your distribution, and add, into your distributions primary .pm module file:

require XSLoader;
XSLoader::load('My::Module', $VERSION);

Normally, the INLINE.h include can be removed, but because we're using some Inline functionality, we need to grab a copy of INLINE.h from somewhere and copy it into the root directory of our distribution so that everything compiles nicely. There's always a copy of it in the _Inline/build/* directory mentioned above. Providing this header file will allow users of your distribution that don't have Inline::C installed to use your module as if they did have it.

SPI bus access, analog in/out on the Raspberry Pi powered by Perl

Well, all of the learning and testing I've done with C, XS, managing bits, reading and understanding hardware datatsheets etc in the last few months is really starting to pay off, with a lot of kudos going out to many Perlers for providing guidance and help with my questions, particularly with XS and C.

We now have reliable, working Perl code to output and receive input analog signals on the Raspberry Pi. This example uses an MCP41010 digital potentiometer for the analog out, and an ADC1015 analog to digital converter for analog in. I still have two different ADCs to write code for, two more models of digital pots, and later this week I should be receiving my DACs (digital to analog converter), my GPS receiver chip, and my MCP3004/8 ADCs.

As a bonus, we also now have direct access to communicate on the SPI bus (as the potentiometer does), with RPi::SPI. I even learned (with help) how to pass a Perl array reference into a C function which gets converted into a C unsigned char *, and how to return a Perl array back from C (I'll write another blog post about these two actions in the coming days).

This setup doesn't really do much, but it's the base of what will eventually allow me to have a Pi in the corner that all it does is pull from github and continuously (and automatically!) run unit tests for the Pi software. However, with true analog output/inputs, there's a lot more a Pi can do.

The schematic and the breadboard layout for the setup.

Code:

use warnings;
use strict;

use RPi::WiringPi;

my $pi = RPi::WiringPi->new;

my $adc = $pi->adc;

my $cs = $pi->pin(18);
my $dpot = $pi->dpot($cs->num, 0);

$dpot->set(0);

print "\nValue, Output %\n\n";

for (0..255){

    if (($_ % 10) != 0 && $_ != 255){
        next;
    }

    $dpot->set($_);

    my $p = $adc->percent(0);

    print "$_/255: $p %\n";

    select(undef, undef, undef, 0.3);
}

print "\n\nOutput % at 127/255\n\n";

$dpot->set(127);

for (0..10){
    print $adc->percent(0) . " %\n";
    select(undef, undef, undef, 0.2);
}

$pi->cleanup;

All it does is switch to different taps (resistor level) on the digital pot which increases/decreases output voltage. The ADC's input pin (A0) is connected directly to the output of the pot, as is the LED, just so I can see visually the changes as well as receive them digitally via the software.

Output:

Value, Output %

0/255: 0.36 %
10/255: 4.24 %
20/255: 8.12 %
30/255: 12.00 %
40/255: 15.88 %
50/255: 19.76 %
60/255: 23.70 %
70/255: 27.58 %
80/255: 31.45 %
90/255: 35.33 %
100/255: 39.21 %
110/255: 43.09 %
120/255: 46.97 %
130/255: 50.85 %
140/255: 54.79 %
150/255: 58.61 %
160/255: 62.48 %
170/255: 66.42 %
180/255: 70.24 %
190/255: 74.12 %
200/255: 77.70 %
210/255: 81.21 %
220/255: 84.91 %
230/255: 88.67 %
240/255: 92.67 %
250/255: 96.97 %
255/255: 99.21 %


Output % at 127/255

49.70 %
49.70 %
49.70 %
49.70 %
49.70 %
49.70 %
49.70 %
49.70 %
49.76 %
49.76 %
49.70 %