Re: What if we could drop archives into @INC?

An article written by brian d foy What if we could drop archives into @INC? got me thinking. Indeed, I've been burned myself by PAR, which is a great instrument, but I don't like that it does a huge unzip every time the application made with it starts. And I don't like even more so that some times PAR forgets to clean up these huge perl trees, eating space in my /tmp.

The problem why it is not that easy to override "use" and "require", is that these are semantic constructs, and are using perl's own file open/close operations, which neither are overridable even with *CORE::GLOBAL:: hack.

Nevertheless, after a couple of days learning about implementation of PerlIO layers and hacking them, the following module was born: File::Redirect. It's basically hacks into open() which is used by perl, and then emulates unix mount semantics.

Warning!!! The only problem is that it all works ONLY if perl was compiled with -DPERL_IMPLICIT_SYS, and that's AFAIU is by default windows perl/strawberry only. I believe one can build such perl on unix but I didn't try.

The simplest case would look such as:


use File::Redirect qw(mount);
mount( 'Simple', { 'a' => 'Hello world!' }, 'simple:') or die;
open F, '<a';
print <F>;
close F;

which indeed does print 'Hello world'. But that's not all! -- we can do this too:


use File::Redirect qw(mount);
mount( 'Simple',
   { '/Foo.pm' => 'package Foo; sub foo { 42 }; 1' }, 
   'simple:');
use lib qw(simple:);
require Foo;
print Foo::foo(), "\n";

and yes, perl is fooled into thinking that 'Foo.pm' is located at path 'simple:/Foo.pm'!

But even that is not all yet. The greatest part is that zip archives can be used, at last! Suppose we make a simple zip file, which looks like this:


$ unzip -l Foo-Bar-0.01.zip
Archive:  Foo-Bar-0.01.zip
  Length     Date   Time    Name
 --------    ----   ----    ----
        0  02-25-12 07:46   Foo-Bar-0.01/
        0  02-25-12 07:47   Foo-Bar-0.01/lib/
        0  02-25-12 07:47   Foo-Bar-0.01/lib/Foo/
       43  02-25-12 07:47   Foo-Bar-0.01/lib/Foo/Bar.pm
 --------                   -------
       43                   4 files

And the only file of interest there is


$ unzip -p Foo-Bar-0.01.zip Foo-Bar-0.01/lib/Foo/Bar.pm 
package Foo::Bar;
sub foo { 42 }
1;

The following code does it, without unpacking on disk:


use File::Redirect::lib ('Zip', 'Foo-Bar-0.01.zip', '/Foo-Bar-0.01/lib');
use Foo::Bar;
print Foo::Bar::foo(), "\n";


So it work kind of like Java's jar files. Kind of, because of dynamic libraries, cannot be treated that way, but all other files can.

To sum it up, it's a pity that the module only works only on some perls, so possibly if there are arguments to use -DPERL_IMPLICIT_SYS on unix builds, we could do something about it?

6 Comments

There's a branch in the PAR SVN repo that loads modules from memory instead of unpacking on disk. But that doesn't work for shared objects. Presumably, with lots and lots of system-specific knowledge and tinkering, even that could be made to work, but I wasn't going to be the one to waste my time on it.

PS: The files remain in /tmp since PAR won't re-extract everything if the cache is warm.

Try: par/branches/fastio$ svn diff -c546

for an example commit that is related.

I think monkey-patching open at such a low level is not the most sensible way to approach this. It's bound to break some things.

Why not put a hook into @INC? It's a long documented interface that has far fewer issues than this.

I took 30 minutes and wrote a module to do this for you.

https://github.com/exodist/archlib

I also uploaded it to cpan, 'archlib' though it will take a couple hours to replicate out to mirrors.

Example:

use archlib 'path/to/archive';
use Module::In::Archive;
use Another::in::Archive;

Enjoy

Leave a comment

About Dmitry Karasik

user-pic I blog about Perl.