Alien Archives

An Organization for Alien::Base

Alien::Base is a system that simplifies writing Alien modules to provide external libraries via CPAN. When I first wrote it, I knew that it was going to be a useful project, and I feel even more strongly about that now.

That said, I feel that I have taken Alien::Base as far as I can on my own.

Several others have offered good and useful suggestions and even patches which have unfortunately languished as I attempted to find time to consider their utility or when my knowledge of the situation was lacking. I have therefore decided to expand the circle to include a few of these contributors. I will remain on the team, where I will attempt to guide (not dictate) the project to continue along the path that I still envision for this work.

Perl5-Alien is a development organization devoted to the continued development of Alien::Base and the Perl 5 Alien ecosystem as a whole.

I hope that this will be a step in the right direction for the Alien ecosystem.

If you would like to submit your name for consideration to join the team, please contact me either on this post, via irc, or email. Of course everyone is welcome to contribute via pull requests or by writing your own Alien module for your favorite external library.

Alien::Base Final Report

I have just sent my grant manager, Makoto Nozaki, my final grant report for Alien::Base. As I have said in the report, it has been slowed recently by my Ph.D. Thesis and Defense (successful!) and the lack of Mac CPANTesters (or at least the lack of reports on my testing modules). TL;DR, Alien::Base is essentially ready, but work still needs to be done, and will continue.

The report is included after the break.

Alien::Base - progress and problems

After a busy Christmas season and being engrossed in my upcoming thesis defense I have found it hard to find too much time to focus on Perl projects. Still Alien::Base has been on my mind, and happily it has been on the minds of others too!

In this report I want to focus first on the high points. Far and away my high point has been that in my apparent absence others have taken up the mantle. I have gotten bug reports and pull requests from preaction, giatorta, amannb, mokko, tobyink and as always productive conversations with David Mertens. It helps in a tough project like this has been to see the excitement of other developers, waiting for me to finish this project.

Several bugs have been fixed, others have been identified. I hope to have another dev release out soon. Further I plan to release a version of Alien::GSL which depends on Alien::Base to CPAN not long after that.

Now for the bad news. While it seems that Alien::Base is coming to a preliminary form of “completion” I have started to notice a disturbing trend coming from CPANtesters. Let me preface this by saying I think CPANtesters might be the killer app of CPAN; it is spectacular and I have said so before. There is a problem though.

Because I wanted to be sure that testing was done with properly installed modules, I created two phony distributions: Acme::Alien::DontPanic which provides a tiny C library of my own creation, and Acme::Ford::Prefect which relies on it. For this reason, when considering the test results for Alien::Base, the results of these modules must be considered as well.

As I have noted before, certain platforms (Mac OSX) need to know the full path of the library’s final install location at build time. The only way I was able to overcome this problem was to have Alien::Base build the library in a temporary location and reference the final location in the build system. Then when the module is installed, the build tool’s install command is issued. The library’s build system therefore avoids the Perl build system. Let me say this a different way, the built library never lands in a blib directory before moving to its final location.

Now I want you to note, that I have taken special care to make sure that Alien::Base refreshes the installation packlist and all-in-all there should have been no adverse effects from this plan. It wasn’t until I started to get failing tests for the packlist handling on Acme::Ford::Prefect that I noticed the problem. While I was able to prevent those problems from being reported, they belied a deeper problem.

This brings us back to CPANtesters. When testing some module, many of the CPANtesters smoke testers don’t actually install the dependent modules. What they do is build each dependent module, then add all the relevant blib directories to the @INC array. This is a clever system, it means that a fresh dependency chain is available for each test, it means that the testing platform is not affected by installing all these modules. Unfortunately its not testing a real install environment. For most people and for most modules this is fine. For Alien::Base as currently constituted however, its a very real distinction.

On platforms where this library path information is not problematic, Alien::Base will work correctly, even under CPANtesters blib scheme. For platforms that suffer from this problem, I believe that Alien::Base will work correctly, but not under the blib scheme. Therefore no dependent modules will be able to successfully dynamically link to the provided C library.

Until Alien::Base-based modules can install and test on the three major platforms, I hesitate to call Alien::Base completed. I think I’m rather close to declaring victory on 5 of those 6 goals, however if CPANtesters testing on Mac is broken, I’m still stuck saying that there is work to be done before any victory may be declared.

Perhaps there is some way that CPAN testers can work with me to avoid this problem, admittedly I haven’t talked to them about this. If anyone with more knowledge on the subject wants to comment below, email me or file a github issue, please do.

Alien::Base Grant Report November

This month featured lots of work in the latter parts of Alien::Base. These improve library detection logic, pkg-config functionality and packlist support.

I was especially pleased to get a bug report filed by bpo regular Toby Inkster. He is writing a provisional Alien::LibXML based on Alien::Base. He noticed some odd behaviors that we are still trying to work out. This feedback led to more improvements than just his, so please keep the bug reports coming!

This month I released three dev versions, the latest being released this evening. I also released Acme::Ford::Prefect again, this time the tests shouldn’t fail on CPANtesters computers who don’t actually install the dependent modules for the one that they are testing. This module is one of the “downstream” modules that are part of the test system for Alien::Base (along with Acme::Alien::DontPanic). These are needed since some of the magic happens during the final installation so tests have to occur AFTER installation of the module in question.

I was saddened this month to learn that Strawberry Perl doesn’t include a Bash interpreter with its build environment. I was under that impression since Strawberry is built on MinGW (compiler) and also includes make, that meant that it bundled MinGW’s MSYS environment. Unfortunately this isn’t true, Strawberry bundles its own make, but not the other tools provided by MSYS, like Bash, and thus it cannot run configure scripts. Oh well, I always planned on pre-built binaries for Windows, guess I’m back to that.

I know I say this regularly, but the system is very close to completion! I’m not going to claim this month again, but sometime very soon!

Alien::Base Perl Foundation Grant Report Oct

This month has been a good one for Alien::Base, and I’m happy to report that I believe that the grant is winding up. We saw a beta release and had a project night during which we hacked on a few issues, but also I was keenly observing people assessment of usability, which I consider key to the success of the project.

I think I have only a few issues to work out before I declare success:

  1. Fix a bug which has crept up on OpenBSD - I am almost certain this is related to my implementation of a parser for pkgconfig files
  2. Investigate a solaris bug reporting from the extended test chain (i.e. Alien::DontPanic and Alien::Ford::Prefect)
  3. Polish up my Alien::GSL distribution and release to CPAN as an example. BTW this was my project on project night, though with all my activity I didn’t get a whole lot done (visibly)

I think that it is conceivable that these last issues can be addressed this month.

Alien::Base Beta Release!

I am happy to announce that Alien::Base (GitHub) has seen a beta release, version 0.001. It seems that my design change that I previously blogged about has indeed fixed (well avoided) the problems that I was having supporting Mac.

This is not to say that Alien::Base is quite completed. While I have released two testing modules which are an Alien:: module (Acme::Alien::DontPanic) and a dependent module (Acme::Ford::Prefect) these are very simple modules. To be sure that the API is flexible enough and that the loader mechanisms are robust enough Alien::Base needs to be used in the wild.

This week’s Chicago/WindyCity.pm meeting, our monthly Project Night, will feature (though not exclusively) creating these modules. I personally will work on porting Alien::GSL to the Alien::Base system. I hope that if you are in the area you will consider attending or if not please attempt to wrap your favorite C library using Alien::Base and let me know how it goes.

Also, please try installing Alien::Base, Acme::Alien::DontPanic and Acme::Ford::Prefect (in that order, though installing Ford should just work), and let me know how that works on your system. I have been developing on Linux and have run some tests on Mac obviously. I have seen some conflicting reports on different Windows/Perl combinations, so this remains to be seen.

I am very happy to have reached this point, and I want to thank the Perl Foundation and my grant manager Makoto Nozaki for their patience with me. This was supposed to be a short project and it has become a much longer one than originally proposed. Hopefully the community will find it useful.

Design considerations for Alien::Base

Anyone who has been following my progress on Alien::Base knows that in the past few months I have been struggling to nail down the final problem, namely Mac’s library name problem. The short story is, on Mac, the library has the full path to the module baked into the library during compilation. My problem is that I don’t tell it the correct path during compilation. Why?

I wanted installing an Alien::Base module to feel as much like installing a regular Perl module. This means following the canonical perl Build.PL, ./Build, ./Build test, ./Build install, incantations that we are all used to. Further I wanted each step to act as normal. To do this, the ./Build step is the one that fetches the library source, configures it compiles it, and then “installs” it to a directory inside the temporary build directory of the Alien::Base-based installer module (this location was set during configure using the --prefix flag). This is so that the user can invoke tests that use the library before installing it to the proper permanent (user-space) location during the ./Build install phase. This is supposed to be the path of least astonishment.

That brings us back to Mac, because the library needs to know its full install path before it is built, and the best I can tell it is the temporary “install” directory, I cannot make the finally installed library function correctly. I have tried to compensate by using tools which can change the location inside the compiled binary library, but there are huge drawbacks to this too, which I will not go into here. Needless to say, it involves parsing and changing directives inside arbitrary Makefiles. It doesn’t work, at least not well enough.

This has lead to stagnation in the Alien::Base project as I have been letting this problem percolate through the old gray matter. Finally, after a few months, I think I might have an idea. Its not my favorite, and it will involve some rewriting, but I think it stays closest to the original goals.

The new plan is now this. ./Build will now fetch, configure and build the library; the --prefix will now point to the final installed location. I will still make attempts to have ./Build test work as expected, at least on most platforms, but this is no longer a priority. I can hear you all crying, “but we need testing!”. Yes and I would like to give it to you, however tests for Alien::Base-based modules essentially come down to testing that the library was properly built (I can still make it run make test or make check) and that it is properly provisioned to Perl. It is this provisioning that is the problem, if I use a temporary/intermediate directory, I can make those tests pass, but its not testing the real world. If I don’t use a temporary directory, you can’t test for provisioning, but it never really was, so at least its not lying to you.

Finally ./Build install would first invoke the usual Perl installation, then run whatever make install-type command that is established for the module, copying the files from the build directory into the designated File::ShareDir-findable location under the Alien::Base-based module’s installed directory tree.

The upside to this procedure is that it will now allow the Makefile to properly handle the location handling, especially for Mac where this is a larger concern than I originally expected.

That said, thank you all for your patience. I really needed this time to consider this second path. I look forward to any comments you may have!

N.B. The source for Alien::Base is available through my GitHub page.

Caught up, finally

Hello again Perl world!

After a wonderful vacation, I came back to discover that I had far more work to do than I had realized. I have only just started to claw out of the heap and arrive at a place where I have had some time for Perl-ing.

First of all I need to apologize. I missed my Grant Report this month. While this is no excuse, there also was nothing to report. I do hope to keep honing in on the few remaining problems that Alien::Base has developed, but I am increasingly believing that a few of my initial assumptions may have been too flawed, possibly requiring a little bit of rewrite. That said, what I really need is someone who has a longer beard than I (metaphorically) to help me understand some Makefile/linking stuff to help me over the hump.

That brings me to some more interesting things. I had almost gotten my additions to vti’s PerlTuts running before my vacation, and now they have finally hit; thats right, live science tutorials in Perl are coming! For now its only a few pages on my plotting extension and PDL constructors, but believe me, that was the hard part.

Finally, this weekend I added up a few more features to Galileo, these include such novel features as deleting pages and easy links to adding users and pages. For those of you who don’t know (many I’m sure), Galileo is my attempt at a fully CPAN installable content management system (website). Its runs on Mojolicious and uses websockets for real-time updating wherever possible. Please take a look and let me know if you have any issues; especially if you use the (still untested, sorry) environment variables for controlling file locations.

Cheers!

Alien::Base Perl Foundation Grant Report Month 5

After another busy month outside of the Perl world, I have gotten a little more time in the last week to work on Alien::Base. I must especially thank fellow WindyCity.pm member David Mertens for working with me on some of the Mac problems involved.

N.B. I also want to thank him for PDL::Graphics::Prima, which made the rest of my $work easier this month! If you need interactive plotting, give it a look! Also Chicagoans, WindyCity.pm is tomorrow, topic: Dancer.

Ok so after much tinkering, I have made enough changes to have the test suite passing on Mac, however, it gives warnings which make me think that while it is “working” in the blib directory, that it won’t work once installed. To this end I have been putting together some “real” Alien::Base modules to be released into the ACME:: namespace (if you care, look for ACME::Alien::DontPanic (which consumes Alien::Base) and ACME::Ford::Prefect which depends on it). To make this a more real test case, I have been working to convert my libdontpanic shared library to the proper autotools chain. I have spent the day learning about autoconf/automake/libtool etc, and let me say, if you never need to learn this, then consider yourself lucky!

Hopefully these test modules will illuminate the real problems in relocating the .dylib files, and perhaps other non-Mac problems too!

Alien::Base Perl Foundation Grant Report Month 4

Well its that time again. Thankfully the news is getting better once more!

Much of my time which is earmarked for Perl went to preparing for my YAPC talks. I’m glad they went so well; thanks to all of you who attended. I did even get a question about Alien::Base during one of the Q&As; I’m glad to know that people are interested.

The news this month hopefully is that Windows is passing tests! Or it should once some windows tests show up. It passes on the only windows dev box that I have access to.

Also I have begun work on the fixes for the Mac problem I described last month. I still would like to figure out some way to ensure that the linker flag -header-pad_max_install_names is passed rather than ensuring a really long build path is used. Perhaps using the Makefile ENV => Variables trick? Of course that assumes your project uses make and doesn’t clobber variables but appends to them.

Future plan:

This Mac problem has stymied me for a while too long, and I think it might be time (now that windows passes) to move forward. Certainly if the Mac host has the library installed Alien::Base should be able to detect it, the problem is only on the installation side and its being looked at. I really need to start getting some real-world feedback. I have heard from several people that they have projects in mind; I think that assuming the windows build tests do pass, its time to move to alpha phase.

During alpha testing I will want people to start creating their own Alien::MyLibrary modules. I especially need to know where the configuration scheme is not general enough. I do also need to know what is confusing in the documentation/examples so I can clear those up too. Of course during alpha testing I am making no promises that the API wont change, so don’t release your dependent Alien:: modules to CPAN just yet, but please do inform me if you make a new github project or branch using Alien::Base, I definitely want to follow those.

I know I have said it before but the official call to arms should come very soon. The released version will be 0.001. Still if you want to get started now, you have my blessing.

Finally I want to mention that my original grant proposal had mentioned being done by now. Obviously this isn’t the case. However, seeing as I am not being paid by the month (or any other timeframe for that matter), I would hope that seeing that the project is still alive and well should be enough to keep my grant open.

Cheers!

About Joel Berger

user-pic As I delve into the deeper Perl magic I like to share what I can.