Progress Report Archives

Alien::Base Final Report

I have just sent my grant manager, Makoto Nozaki, my final grant report for Alien::Base. As I have said in the report, it has been slowed recently by my Ph.D. Thesis and Defense (successful!) and the lack of Mac CPANTesters (or at least the lack of reports on my testing modules). TL;DR, Alien::Base is essentially ready, but work still needs to be done, and will continue.

The report is included after the break.

Alien::Base - progress and problems

After a busy Christmas season and being engrossed in my upcoming thesis defense I have found it hard to find too much time to focus on Perl projects. Still Alien::Base has been on my mind, and happily it has been on the minds of others too!

In this report I want to focus first on the high points. Far and away my high point has been that in my apparent absence others have taken up the mantle. I have gotten bug reports and pull requests from preaction, giatorta, amannb, mokko, tobyink and as always productive conversations with David Mertens. It helps in a tough project like this has been to see the excitement of other developers, waiting for me to finish this project.

Several bugs have been fixed, others have been identified. I hope to have another dev release out soon. Further I plan to release a version of Alien::GSL which depends on Alien::Base to CPAN not long after that.

Now for the bad news. While it seems that Alien::Base is coming to a preliminary form of “completion” I have started to notice a disturbing trend coming from CPANtesters. Let me preface this by saying I think CPANtesters might be the killer app of CPAN; it is spectacular and I have said so before. There is a problem though.

Because I wanted to be sure that testing was done with properly installed modules, I created two phony distributions: Acme::Alien::DontPanic which provides a tiny C library of my own creation, and Acme::Ford::Prefect which relies on it. For this reason, when considering the test results for Alien::Base, the results of these modules must be considered as well.

As I have noted before, certain platforms (Mac OSX) need to know the full path of the library’s final install location at build time. The only way I was able to overcome this problem was to have Alien::Base build the library in a temporary location and reference the final location in the build system. Then when the module is installed, the build tool’s install command is issued. The library’s build system therefore avoids the Perl build system. Let me say this a different way, the built library never lands in a blib directory before moving to its final location.

Now I want you to note, that I have taken special care to make sure that Alien::Base refreshes the installation packlist and all-in-all there should have been no adverse effects from this plan. It wasn’t until I started to get failing tests for the packlist handling on Acme::Ford::Prefect that I noticed the problem. While I was able to prevent those problems from being reported, they belied a deeper problem.

This brings us back to CPANtesters. When testing some module, many of the CPANtesters smoke testers don’t actually install the dependent modules. What they do is build each dependent module, then add all the relevant blib directories to the @INC array. This is a clever system, it means that a fresh dependency chain is available for each test, it means that the testing platform is not affected by installing all these modules. Unfortunately its not testing a real install environment. For most people and for most modules this is fine. For Alien::Base as currently constituted however, its a very real distinction.

On platforms where this library path information is not problematic, Alien::Base will work correctly, even under CPANtesters blib scheme. For platforms that suffer from this problem, I believe that Alien::Base will work correctly, but not under the blib scheme. Therefore no dependent modules will be able to successfully dynamically link to the provided C library.

Until Alien::Base-based modules can install and test on the three major platforms, I hesitate to call Alien::Base completed. I think I’m rather close to declaring victory on 5 of those 6 goals, however if CPANtesters testing on Mac is broken, I’m still stuck saying that there is work to be done before any victory may be declared.

Perhaps there is some way that CPAN testers can work with me to avoid this problem, admittedly I haven’t talked to them about this. If anyone with more knowledge on the subject wants to comment below, email me or file a github issue, please do.

Alien::Base Grant Report November

This month featured lots of work in the latter parts of Alien::Base. These improve library detection logic, pkg-config functionality and packlist support.

I was especially pleased to get a bug report filed by bpo regular Toby Inkster. He is writing a provisional Alien::LibXML based on Alien::Base. He noticed some odd behaviors that we are still trying to work out. This feedback led to more improvements than just his, so please keep the bug reports coming!

This month I released three dev versions, the latest being released this evening. I also released Acme::Ford::Prefect again, this time the tests shouldn’t fail on CPANtesters computers who don’t actually install the dependent modules for the one that they are testing. This module is one of the “downstream” modules that are part of the test system for Alien::Base (along with Acme::Alien::DontPanic). These are needed since some of the magic happens during the final installation so tests have to occur AFTER installation of the module in question.

I was saddened this month to learn that Strawberry Perl doesn’t include a Bash interpreter with its build environment. I was under that impression since Strawberry is built on MinGW (compiler) and also includes make, that meant that it bundled MinGW’s MSYS environment. Unfortunately this isn’t true, Strawberry bundles its own make, but not the other tools provided by MSYS, like Bash, and thus it cannot run configure scripts. Oh well, I always planned on pre-built binaries for Windows, guess I’m back to that.

I know I say this regularly, but the system is very close to completion! I’m not going to claim this month again, but sometime very soon!

Alien::Base Perl Foundation Grant Report Oct

This month has been a good one for Alien::Base, and I’m happy to report that I believe that the grant is winding up. We saw a beta release and had a project night during which we hacked on a few issues, but also I was keenly observing people assessment of usability, which I consider key to the success of the project.

I think I have only a few issues to work out before I declare success:

  1. Fix a bug which has crept up on OpenBSD - I am almost certain this is related to my implementation of a parser for pkgconfig files
  2. Investigate a solaris bug reporting from the extended test chain (i.e. Alien::DontPanic and Alien::Ford::Prefect)
  3. Polish up my Alien::GSL distribution and release to CPAN as an example. BTW this was my project on project night, though with all my activity I didn’t get a whole lot done (visibly)

I think that it is conceivable that these last issues can be addressed this month.

Caught up, finally

Hello again Perl world!

After a wonderful vacation, I came back to discover that I had far more work to do than I had realized. I have only just started to claw out of the heap and arrive at a place where I have had some time for Perl-ing.

First of all I need to apologize. I missed my Grant Report this month. While this is no excuse, there also was nothing to report. I do hope to keep honing in on the few remaining problems that Alien::Base has developed, but I am increasingly believing that a few of my initial assumptions may have been too flawed, possibly requiring a little bit of rewrite. That said, what I really need is someone who has a longer beard than I (metaphorically) to help me understand some Makefile/linking stuff to help me over the hump.

That brings me to some more interesting things. I had almost gotten my additions to vti’s PerlTuts running before my vacation, and now they have finally hit; thats right, live science tutorials in Perl are coming! For now its only a few pages on my plotting extension and PDL constructors, but believe me, that was the hard part.

Finally, this weekend I added up a few more features to Galileo, these include such novel features as deleting pages and easy links to adding users and pages. For those of you who don’t know (many I’m sure), Galileo is my attempt at a fully CPAN installable content management system (website). Its runs on Mojolicious and uses websockets for real-time updating wherever possible. Please take a look and let me know if you have any issues; especially if you use the (still untested, sorry) environment variables for controlling file locations.


Alien::Base Perl Foundation Grant Report Month 5

After another busy month outside of the Perl world, I have gotten a little more time in the last week to work on Alien::Base. I must especially thank fellow member David Mertens for working with me on some of the Mac problems involved.

N.B. I also want to thank him for PDL::Graphics::Prima, which made the rest of my $work easier this month! If you need interactive plotting, give it a look! Also Chicagoans, is tomorrow, topic: Dancer.

Ok so after much tinkering, I have made enough changes to have the test suite passing on Mac, however, it gives warnings which make me think that while it is “working” in the blib directory, that it won’t work once installed. To this end I have been putting together some “real” Alien::Base modules to be released into the ACME:: namespace (if you care, look for ACME::Alien::DontPanic (which consumes Alien::Base) and ACME::Ford::Prefect which depends on it). To make this a more real test case, I have been working to convert my libdontpanic shared library to the proper autotools chain. I have spent the day learning about autoconf/automake/libtool etc, and let me say, if you never need to learn this, then consider yourself lucky!

Hopefully these test modules will illuminate the real problems in relocating the .dylib files, and perhaps other non-Mac problems too!

Alien::Base Perl Foundation Grant Report Month 4

Well its that time again. Thankfully the news is getting better once more!

Much of my time which is earmarked for Perl went to preparing for my YAPC talks. I’m glad they went so well; thanks to all of you who attended. I did even get a question about Alien::Base during one of the Q&As; I’m glad to know that people are interested.

The news this month hopefully is that Windows is passing tests! Or it should once some windows tests show up. It passes on the only windows dev box that I have access to.

Also I have begun work on the fixes for the Mac problem I described last month. I still would like to figure out some way to ensure that the linker flag -header-pad_max_install_names is passed rather than ensuring a really long build path is used. Perhaps using the Makefile ENV => Variables trick? Of course that assumes your project uses make and doesn’t clobber variables but appends to them.

Future plan:

This Mac problem has stymied me for a while too long, and I think it might be time (now that windows passes) to move forward. Certainly if the Mac host has the library installed Alien::Base should be able to detect it, the problem is only on the installation side and its being looked at. I really need to start getting some real-world feedback. I have heard from several people that they have projects in mind; I think that assuming the windows build tests do pass, its time to move to alpha phase.

During alpha testing I will want people to start creating their own Alien::MyLibrary modules. I especially need to know where the configuration scheme is not general enough. I do also need to know what is confusing in the documentation/examples so I can clear those up too. Of course during alpha testing I am making no promises that the API wont change, so don’t release your dependent Alien:: modules to CPAN just yet, but please do inform me if you make a new github project or branch using Alien::Base, I definitely want to follow those.

I know I have said it before but the official call to arms should come very soon. The released version will be 0.001. Still if you want to get started now, you have my blessing.

Finally I want to mention that my original grant proposal had mentioned being done by now. Obviously this isn’t the case. However, seeing as I am not being paid by the month (or any other timeframe for that matter), I would hope that seeing that the project is still alive and well should be enough to keep my grant open.


Alien::Base Perl Foundation Grant Report Month 3

This month’s work on Alien::Base started really exciting. I had tracked down several bugs and was honing in on full Linux compatibiliy. Turns out much of my testing problems had been in the test suite, wherein I mimiced make/Makefile with a perl script (in the name of Xplatform) but made some incorrect assumptions.

The problem came when I went to look at OS X. The problem I have found is that Mac bakes the full path to the library into the library itself during compiling/linking. It can be changed manually as long as the new path is shorter than the old one! Yes there are work-arounds but they would involve delving into the library’s Makefile, which I have been ardently trying to avoid (opens a pandoras box). The other mechanism would be to target the library to install in the final place directly, but this breaks the usual workflow of installing a Perl module, in which perl Build.PL and ./Build do not do anything permenant, waiting until ./Build install to move the files to their final places.

This problem cropped up a few weeks ago and since then my work on Alien::Base has been residing in the grey-matter, hoping a solution would present itself. I don’t want to say “forget Mac” because the whole point of Alien is to be as cross-platform as possible, still I keep coming back to thinking that perhaps an early release might have to forgo Mac initially.

My research has picked up, some of you may have seen my PerlGSL and related modules arrive this month; these modules are not purely a diversion from Alien::Base but rather they have been critical in some simulations I am developing. These and my thesis have taken some time from Alien::Base but I knew that (and disclosed it) going into the grant.

I am targeting a Linux release before next month’s grant report as most of the API seems sufficient, though I am reluctant to release while the Mac issue is outstanding like this. Hopefully the Perl Foundation will be patient with me as I deal with these pressing matters both inside and outside of the Alien::Base project.

Alien::Base Perl Foundation Grant Report Month 2

After last month’s breakneck development pace, I knew this month wouldn’t be as gratifying, and indeed it turned into quite a slog.

This month involved lots of little bug fixes, posting dev releases to CPAN, then waiting for test results from CPANtesters. As a side note, there are a larger number of reports coming from Solaris and BSD than I would have expected. Sadly one of the bugs that still hasn’t been sorted out is this recurring Solaris bug when changing working directory. It would appear that I am going to have to find a Solaris box or VirtualBox appliance, since waiting for test results for every fix attempt would take far too long.

I do have some nice things to report. I have received help from fellow member David Mertens and new contributor JT Palmer this month, so thanks guys! Next I have just pushed a new dev release which (again) changes the mechanism of dynamic loading; this one more reliant on DynaLoader’s facilities rather than munging ENV variables. This seems to be more platform independent, or I should say, as platform independent as p5p has written into it. I have much faith in them! Preliminary results seem promising. David had been having problems with a candidate Alien:: module that he is writing and this release seems to have fixed it. I hope to see Darwin and Win32 start passing too (gasp).

With that said, I am still targeting more documentation and a basic testing framework before a 0.001 alpha release. Seems like I’m always saying this, but I hope this will be coming shortly.

Fork Alien::Base on Github and note that the 0.000_009 dev release is still contained in the dlopen branch. I will merge it once the tests seem to bear it out.

Alien::Base Perl Foundation Grant Report Month 1

Once again I would like the thank the Perl Foundation for supporting me in my effort to provide a mechanism to ease the creation of Alien:: modules. Further I’d like to thank the many Perlers who have commented in various places that this project is of interest and that they are looking forward to providing that Alien:: module that they have always meant to write. This is exactly the response that I had hoped to receive.

Down to the details. This month I did a lot of work on Alien::Base; partially do the excitement about the grant and partially because our scientific camera was out for repairs, thus not much science going on in the lab. I hope to keep the pace high, but looking over the git log I’m not sure that they can be this productive! I’ll list some high points:

  • A::B now relies on M::B to handle most of the attributes. Originally I was pulling out my hash keys before creating the object. The few that I still do parse manually I do because there are states that M::B expects prior to the object being created, and I must ensure that those states exist.
  • A::B now includes a simple pkgconfig file parser and storage/munging engine. This allows for A::B to know what cflags and libs to provide and which -L directories to include even after M::B moves the files to their final location. I am considering using mnunberg’s new PkgConfig module to do the parsing, but in truth, the spec doesn’t seem very difficult (famous last words? let me know in the comments)
  • Removed an awkward system for supplying defaults for all repositories and building repository objects from them. The new engine for this is much simpler and allows for true constructors for the repository objects. This is necessary if my goal of presenting a fully subclassable module
  • Possibly the most important item: use Alien::MyModule; now provides the location of the library to the Perl’s run-time loader via LD_RUN_PATH (or PATH on windows). I may need to expand this for other platforms; I’m not familiar with this stuff on any platform but Linux, though my goal has always been to be as x-platform as possible
  • Wrote lots of documentation and tests. Of course there is plenty more to go, but I think that a reasonably competent CPAN author could write an Alien:: module using only these documents. See especially: A::B::Authoring and A::B::ModuleBuild::API.
  • Created an initial dev release, 0.000_001 and given the testers results, two further releases; now at 0.000_003.

Still on my list:

  • Fix a recurring bug that the testers see. It seems to always happen on Solaris, but also appears occasionally on other platforms.
  • Provide a few inheritable tests so that Alien:: authors can have a backbone test provided for them. I expect this will be something in a Test::Class kind of implementation. However the abstractions make this test hard to test, so I’m working on it.

Finally I want to share a little story. At just the right time, I happened to notice that Richard Simões had started an Alien::Nettle project on his GitHub account. Just for fun, I decided to see how easy it would be to write my own Alien::Nettle for this nettle library that I had never heard of, but which seemed to meet my mental image of a good candidate for the Alien::Base system. The project to me all of about 5 minutes and worked successfully! I contacted Richard to mention it and he was quite excited.

This has been added to the examples directory (in his name). There are other examples too, including my own pet Alien::GSL. Take a look to see how easy this can be.

Do you have an Alien:: project that you wish you could write? Once I knock out this bug, I will release an alpha version, probably 0.001 and I will issue a call-to-arms. The reason for this is two-fold.

  1. I want to hear what problems you have in writing your Alien:: module. Where is the framework not flexible enough? How is it confusing to you? Alien::Base has no purpose without users, I want to have you involved sooner rather than later.
  2. I want my actual release (1.000) to be accompanied by a host of other releases of YOUR Alien:: projects. Hopefully coordinated on the same day. What could be a better demonstration of a new easy mechanism for library support?

Finally, just wanted to mention that the project is available on GitHub - including those examples :-) Please, take a peek and comment either there or here, email me at joel dot a dot berger at gmail, or even chat me up on #alien, which I have created to support the authors after my call-to-arms.


About Joel Berger

user-pic As I delve into the deeper Perl magic I like to share what I can.