Practical FFI with Platypus
At YAPC::NA 2014 I talked about FFI and Perl. FFI is an alternative to XS that I think is worthy of consideration. My talk was well attended, I think primarily because I jokingly subtitled my talk "Never Need To Write XS Again". So there is a market for this idea. I mostly talked about FFI::Raw, which was a great way to experiment with FFI and to write real live CPAN modules with FFI right then and there. The question of performance inevitably came up, so at the Pittsburgh Perl Workshop last year I talked about that.
Essentially I think the performance gains you get from a well tuned XS extension are not always justified by the amount of time that you spend debugging them. I also think that FFI more easily ports your skills to other technologies and lends itself more easily to a bright future for Perl in which there are multiple implementations, some of which do not need to support XS. All that being said, there were some tweaks that could be made in order to make FFI faster on Perl, so I wrote a prototype code named Platypus to experiment with that idea. At Pittsburgh I showed off the performance benchmarks for my prototype, which were very good in comparison to FFI::Raw and not terrible in comparison to XS and Inline::C.
I've now reworked FFI::Platypus and released it to CPAN. Feedback is welcome. Very briefly benefits include:
- Practical bindings to libffi to write stand alone scripts and CPAN modules today in 2015
- Faster invocation through real xsubs rather than object oriented method dispatch
- Support for types not directly provided by libffi like pointers and arrays
- API for creating custom types (an analogue to typemaps in XS)
There are a number of examples included with the FFI::Platypus distribution. Check it out if I have piqued your interest. Constructive feedback and pull requests are welcome.
Dear Graham Ollis,
FFI::Platypus is awesome :) I am trying it now.
I could install in Windows 7 + Active Perl + cpan with some manual setting of %PKG_CONFIG_PATH% environment variable pointing to libffi downloaded via Alien::FFI.
Not a big problem :) In linux it works like a charm.
I feel the documentation can be improved in the view of people with limited C knowledge.
How do I create C structure and pass to a lib function using FFI::Platypus? What exactly it provides more than FFI::Raw? Just for my better understanding, I have asked the above questions.
Thanks for your work.
Thanks for your feedback Sam.
Compared to FFI::Raw, Platypus allows you to easily address some common C patterns for passing arguments like:
- pointers to native types (int, float, etc) for pass by reference
- fixed arrays of native types
- custom types for anything that Platypus core does not (yet) support
These can be done in FFI::Raw, but not without extra objects or wrapper functions. Big motivation for Platypus was the type system, which I realized was rather limited in FFI::Raw when I was writing Archive::Libarchive::FFI. The goal is to make the most common things easy, and provide tools for writing reusable types for the things that do not belong in the Platypus core.
Platypus also allows you to attach a function as a real Perl sub:
This turns out to be much faster when your function is called repeatedly than creating a sub around your FFI::Raw object and calling it:
Your comment about the documentation betrays my background as a C programmer :) I encourage pull requests to improve the documentation, or just open an issue with pointers to what is especially confusing. I would love to see some tutorials or FAQs for interacting with other compiled languages like Fortran, Rust, Go, assembly.
As far as structures, Platypus does not address it by itself, but there are a few ways to deal with them already today:
- Perl's pack/unpack
- Convert::Binary::C
- Inline::Struct
- bundle your extension with a ffi directory or use FFI::TinyCC to write acessors in C and import them into your namespace with Platypus.
All of these have pluses and minuses. What ever method you choose you need to get a pointer to the struct and pass it in as an opaque type. The documentation should include some examples, I will get to that soon. In the longer term, I'd like to see some easy way to do at least simple structs in Platypus though. Ruby-FFI has a structure layout API that could be worth cribbing from.
The issue with Alien::FFI and windows should be solved soon. There is a dev version 0.04_01 that works with older strawberries that do not come bundled with libffi, and I suspect that the problem with ActiveState Perl is similar. If it isn't solved when 0.05 goes to CPAN please open an issue in the project github.
Also I am planning on posting some more entries about FFI + Platypus here, about practical use of Platypus so keep an eye out for that. I was going to write one on pointers, but I think that one on structs is probably needed more.
Thanks for the explanation Graham Ollis, I will do experiments with Structure + FFI::Platypus.
Dear Graham Ollis,
I am trying to use a Structure with FFI::Platypus.
OS: Windows 7, x64
Perl: Active Perl 5.20, x86
I have written simple code to get Windows System time. This code is not working.
I am not sure where I have missed it. If this works, It could be an example of how to use struct with FFI::Platypus.
#!perl
use strict;
use warnings;
use FFI::CheckLib;
use FFI::Platypus;
use Convert::Binary::C;
use Data::Dumper;
use Data::Hexdumper;
#Get the system time using Kernel32.dll
#find the Kernel32.dll
my $libPath = find_lib(lib=>'Kernel32');
#Create FFI Object
my $ffiObj = FFI::Platypus->new();
$ffiObj->lib($libPath);
#Import the GetLocalTime function
$ffiObj->attach('GetLocalTime',['opaque'],'void');
#Define SYSTEMTIME Struct as per https://msdn.microsoft.com/en-us/library/windows/desktop/ms724950(v=vs.85).aspx
#As per, C:\MinGW\include\windef.h, WORD id unsigned short
my $c = Convert::Binary::C->new->parse(
struct SYSTEMTIME {
unsigned short wYear;
unsigned short wMonth;
unsigned short wDayOfWeek;
unsigned short wDay;
unsigned short wHour;
unsigned short wMinute;
unsigned short wSecond;
unsigned short wMilliseconds;
};
ENDC
my $dateStruct = {
wYear=>0,
wMonth=>0,
wDayOfWeek=>0,
wDay=>0,
wHour=>0,
wMinute=>0,
wSecond=>0,
wMilliseconds=>0,
};
my $packed = $c->pack('SYSTEMTIME', $dateStruct);
#Call the function by passing the structure reference
GetLocalTime(\$packed);
if (defined ($packed))
{
print "\n Got the time";
#Unpack the structure
}
else
{
print "\n Something is wrong";
}
exit 0;
What am I doing wrong?
Try this:
I like the way that you did that though, I think it is more intuitive than the existing way, so I might add it as a Platypus feature:
https://github.com/plicease/FFI-Platypus/issues/22
Great :). It works. You rock Graham Ollis.
#!perl
use strict;
use warnings;
use FFI::CheckLib;
use FFI::Platypus;
use Convert::Binary::C;
use Data::Dumper;
use Data::Hexdumper;
#Get the system time using Kernel32.dll
#find the Kernel32.dll
my $libPath = find_lib(lib=>'Kernel32');
#Create FFI Object
my $ffiObj = FFI::Platypus->new();
$ffiObj->lib($libPath);
#Import the GetLocalTime function
$ffiObj->attach('GetLocalTime',['opaque'],'void');
#Define SYSTEMTIME Struct as per https://msdn.microsoft.com/en-us/library/windows/desktop/ms724950(v=vs.85).aspx
#As per, C:\MinGW\include\windef.h, WORD id unsigned short
my $c = Convert::Binary::C->new->parse(
struct SYSTEMTIME {
unsigned short wYear;
unsigned short wMonth;
unsigned short wDayOfWeek;
unsigned short wDay;
unsigned short wHour;
unsigned short wMinute;
unsigned short wSecond;
unsigned short wMilliseconds;
};
ENDC
my $dateStruct = {
wYear=>0,
wMonth=>0,
wDayOfWeek=>0,
wDay=>0,
wHour=>0,
wMinute=>0,
wSecond=>0,
wMilliseconds=>0,
};
my $packed = $c->pack('SYSTEMTIME', $dateStruct);
my $pointerToDateStruct = $ffiObj->cast('string','opaque',$packed); #Get the pointer to the Struct
#Call the function by passing the structure reference
GetLocalTime($pointerToDateStruct);
if (defined ($packed))
{
print "\n Got the time";
#Unpack the structure
print hexdump(data=>$packed);
my $sysDate = $c->unpack('SYSTEMTIME', $packed);
print "\n SYSTEM TIME:",Dumper($sysDate);
}
else
{
print "\n Something is wrong";
}
exit 0;
Is it not simply enough to pass the $packed var's reference like \$packed? Why should the explicit cast is needed?
Stefan Seifert (aka nine) is due to show large and complex Perl 5 CPAN offerings like Catalyst and various XS modules working nicely with Perl 6 via his Inline::Perl5 module at FOSDEM.
His 40 minute talk is scheduled for Saturday: https://fosdem.org/2015/schedule/event/modules_ecosystem_perl6/
Some folk think the various things being shown at FOSDEM, including and maybe especially nine's stuff, may be a game changer viz-a-viz folks' perspectives on Perl 6 and Perl in general. As such I expect there to be related buzz about this this weekend and next week.
My question for you is: does your FFI stuff also play nicely with Inline::Perl5 (or the underlying NativeCall and Inline tech)?
I don't have time to test this myself until next week but maybe you or someone else can talk with nine before his talk? Maybe he can mention how and why your stuff does (or doesn't) work with his stuff?
If someone does have time then I suggest they drop in on the IRC channel #perl6 on freenode and explain what they're doing and ping 'nine'.
I'm sorry I didn't see your comment until just now. The Inline::Perl5 stuff sounds exciting I will have to dig in and find out more about it.
As far as compatibility, I don't see any reason that it Inline::Perl5 shouldn't be able to use Platypus, if it supports a sufficiently broad subset of XS. The only exotic XS thing that it does is uses CvXSUBANY which is undocumented but seems to be an open secret among those in the know about XS. It would be an interesting experiment to actually try it.
I looked at NativeCall some time ago, just the documentation mind you I haven't played with Perl6. My experience has been good porting existing FFI modules from ruby to Perl 5, and I don't see that porting from Perl 5 to Perl 6 in a similar fashion should be too hard.
Ultimately I'd like to see a common spec for FFI / native call stuff, something like a .h file for FFI that could be shared between different languages.