Diab Jerius

I've been working through the design and implementation (using Moo) of a module (IPC::PrettyPipe) which implements some backend functionality via plugins. The nature of the module isn't that important; suffice it to say that there's a rendering backend and an execution backend.

I've previously implemented pluggable backends using Module::Pluggable and kin, but since I'm using Moo for this module I thought I'd investigate how its capabilities enabled (or constrained) pluggability.

(A point of note: Since in Moo class construction is done after the compilation stage, technically one could do all sorts of sly things, but I'd like to avoid too much sausage making. When I write "runtime", I mean after classes have been constructed.)

The main guiding principles for this module are

• Ease of use: I'd like this to be accessible to the casual user. There shouldn't be a lot of setup or boilerplate code. The user shouldn't need to know what's under the hood or that it's implemented in Moo. I'd like to avoid forcing the user to create classes.

• Loose coupling: The rendering and execution functionality should be easily customized. For example, I'd like to be able to use a variety of the execution pipeline modules on CPAN (such as IPC::Run, IPC::Pipeline, IPC::Exe). I'll have to write simple adaptors for those, but the IPC::PrettyPipe interface should make it easy to choose between them.

More technical requirements are:

• Backends must be selectable at runtime.

• The user should be able to choose a backend both during and after the construction of an IPC::PrettyPipe object.

• The user must be able to change a backend after one has been selected.

• There must be access to the backends to leverage any unique functionality. While IPC::PrettyPipe will provide an interface to common functionality, if a backend has special capabilities, it should be possible to access them.

What follows is a brief examination of the different approaches I explored.

Roles

Roles seem like a natural approach as they transparently

• provide direct access to the IPC::PrettyPipe object by the backend routines;
• merge unique backend functionality into the object;

For example, in addition to the run method, IPC::Run provides the start, pump, and finish methods for interacting with the pipeline while it is running. The adaptor for IPC::Run could compose those methods into the derived class. If there are any extra attributes used to customize the backend's behavior, these are composed in as well.

Roles are typically applied during class construction, but they can also be applied dynamically at runtime. Once a role is composed into a class or object, it can't be changed, so it's not possible to meet the requirement to change a backend. It's still interesting to see how they impact the implementation, as that behavior is often not required.

"Static" Composition

Static composition forces selection of the backends during class construction, so it doesn't meet the runtime selection criteria. It does however provide a baseline to compare against for implementation and interface complexity.

Here, IPC::PrettyPipe is used as a base class and the render and execution functionality are composed into the derived class:

package MyPipe;
use Moo;
extends 'IPC::PrettyPipe';

# add the renderer
with 'IPC::PrettyPipe::Render::Template::Tiny';

# add the execution backend
with 'IPC::PrettyPipe::Execute::IPC::Run';

Dynamic Class Composition

Moo::Role (via Role::Tiny) provides the ability to dynamically create composed classes:

use Moo::Role ();

my $class = Moo::Role->create_class_with_roles( 'IPC::PrettyPipe',
    'IPC::PrettyPipe::Render::Template::Tiny',
    'IPC::PrettyPipe::Execute::IPC::Run' );

thus allowing runtime selection of the backends. Without some interface sugar this is too complex of an API to expose. It's just as easy to compose into an object and provide a natural interface to it, which is what the next section discusses.

Dynamic Object Composition

Role::Tiny allows one to compose a role into an existing object. This has the same drawback as dynamic class composition, but the benefit is that it's easy to implement selection of roles via methods, leading to a more traditional API:

my $pipe = IPC::PrettyPipe->new;
$pipe->renderer( 'Template::Tiny' );
$pipe->executor( 'IPC::Run' );

This is implemented in the base class as:

use Moo::Role ();
use Module::Path qw[ module_path ];
use Module::Runtime qw[ compose_module_name ];

sub renderer { $_[0]->_backend_compose( Render => $_[1] }
sub executor { $_[0]->_backend_compose( Execute => $_[1] }

# generic dynamic backend composition
sub _backend_compose {

    my ( $self, $type, $req ) = @_;

    my $module = compose_module_name( "IPC::PrettyPipe::$type", $req );

    croak( "unknown $type ($req => $module)\n" )
      unless defined module_path( $module );

    Moo::Role->apply_roles_to_object( $self, $module );

    return;
}

For the sake of clarity the above implementation doesn't allow specifying backends during object construction; here's the code which provides that functionality:

has _executor => (
    is => 'rwp',
    predicate => 1,
    init_arg => 'executor',
    trigger => sub { $_[0]->_backend_compose( Execute => $_[1] },
);

sub executor { $_[0]->_set__executor( $_[1] )
               if defined $_[1] && ! $_[0]->_has_executor }

I'm using the trigger to ensure that composition is localized to a single spot in the code. (The double underscore in _set__executor is because Moo retains the leading underscore in _executor when it creates the private writer.)

So now, this is possible:

my $pipe = IPC::PrettyPipe->new( renderer => 'Template::Tiny',
                                 executor  => 'IPC::Run'
                               );

There's still no means of changing the backends, but the implementation and exposed interface are quite clean.

Method Delegation

Moving the backends into their own objects provides some flexibility by allowing them to be changed on the fly, but makes exposing any unique functionality more complex.

A complete separation of backend objects from the pipeline object might look like this:

my $pipe = IPC::PrettyPipe->new;

my $renderer = IPC::PrettyPipe::Render::Template::Tiny->new();
my $executor = IPC::PrettyPipe::Execute::IPC::Run->new();

$renderer->render( $pipe );

That's horrible, but very flexible. A cleaner approach would be for the IPC::PrettyPipe object to contain the backend objects, and use method delegatation to invoke the appropriate methods on the contained backend objects.

IPC::PrettyPipe would provide methods to specify the backends, just as in the dynamic object composition pattern:

my $pipe = IPC::PrettyPipe->new;

my $renderer =
    IPC::PrettyPipe::Render::Template::Tiny->new( pipe => $pipe );
my $executor =
    IPC::PrettyPipe::Execute::IPC::Run->new( pipe => $pipe );

$pipe->renderer( $renderer );
$pipe->executor( $executor );

Here the IPC::PrettyPipe object is passed to the backend constructors; this is necessary if native Moo method delegation is used (more on that later).

To cut down on the boilerplate, the interface should also allow specification of just the class:

my $pipe = IPC::PrettyPipe->new;

$pipe->renderer( 'Template::Tiny' );
$pipe->executor( 'IPC::Run' );

It's also a requirement that the backends be specifiable during object construction:

my $pipe = IPC::PrettyPipe->new( renderer => 'Template::Tiny',
                                 executor => 'IPC::Run' );

Finally, the backend object (however it's created) must be directly retrievable so that any unique functionality is exposed:

my $renderer = $pipe->renderer;

The combination of the dual nature of the backend specifications and the requirement for specification during and after object construction added more complexity than I'd initially expected, as will be seen in the following discussion.

Attribute Coercion

Attribute coercion seems the perfect solution to handle the dual nature of the backend specification. The coercion code is run whenever the attribute is set.

use Safe::Isa;

has executor => (
    is => 'rw',

    coerce => sub {
        # create the object if the attribute can't run()
        $_[0]->$_can( 'run' )
             ? $_[0]
             : _backend_factory( Execute => $_[0] ) },

    handles => [ 'run' ],
    lazy => 1,
    default => sub { 'IPC::Run' }
);

where _backend_factory is a generic factory for backends (see the Appendix).

Unfortunately, this code won't work. Backend objects constructed by the coercion routine know nothing about the pipeline object:

1. delegated methods aren't passed any extra parameters to the delegated object;
2. The coercion coderef isn't passed a handle to the object, so can't pass it to the backend constructor.

I could explicitly create delegating methods to work around the first behavior:

sub run { $_[0]->executor->run( $_[0] ) }

but that destroys the elegance of the built in method delegation.

Additionally, it doesn't help with the situation where a backend object needs access to the pipeline object during its construction.

Triggers are a wonderful thing

The mechanism which constructs the backend object must

1. get a reference to the IPC::Pipeline object; and
2. get called whenever the attribute is changed

That would seem to be a trigger. The trigger would decide whether an object needs to be constructed, and if so, would update the attribute with the object. _backend_factory is now a method which passes along the IPC::PrettyPipe object to the backend object's constructor:

has executor => (
    is       => 'rw',
    trigger => sub {
                my ( $self, $val ) = @_;
                $self->executor( $self->_backend_factory( Execute => $val ) )
                    unless $val->$_can( 'run' );
    },
    handles => [ 'run' ],
    lazy => 1,
    default => sub { $_[0]->_backend_factory( Execute => 'IPC::Run' ) },
);

By their nature triggers are called whenever the attribute is set, so if we set the attribute in a trigger we descend into infinite recursion hell as the accessor calls the trigger which calls the accessor which calls the trigger...

(Technically this particular example won't, as _backend_factory will return an object which passes the $_can('run') test so that the next call to the trigger will return without setting the attribute. But it is not necessarily a design pattern I'd generalize.)

There's still some code duplication: with current Moo (1.000007) setting an attribute value via its default subroutine does not invoke a trigger, so the default for executor has to call _backend_factory. It'd be nicer if I could specify

default => sub { 'IPC::Run' }

Note that the backend object is now created when the executor attribute is read or written. I'd prefer it to be only when read, but this approach doesn't allow that.

A Trigger with a proxied attribute

Continuing in this vein, to avoid possible run-away recursion one could make the executor attribute a proxy for a hidden attribute which does the work:

has executor => (
    is       => 'rw',
    trigger  => sub {
            my ( $self, $backend ) = @_;
            my $executor = $backend->$_can( 'run' )
                         ? $backend
                         : $self->_backend_factory( Execute => $backend );
            $self->_set__executor( $executor );
        },
);

has _executor => (
    is => 'rwp',
    init_arg => undef,
    handles => [ 'run' ],
    lazy => 1,
    default => sub { $_[0]->_backend_factory( Execute => 'IPC::Run' ) },
);

All internal code would use _executor. This has the same pseudo-lazy quality of object construction as the previous example, and still has the duplicated calls to _backend_factory. This is safe from infinite recursion, but unfortunately it's now impossible to guarantee retrieval of the actual backend object.

If the executor attribute is never set, the _executor attribute will get a default value, but that isn't passed back to the executor attribute. And if executor was passed a module name, its accessor returns that name, not the actual backend object generated by the trigger. The key here is that the value in _executor is always correct. We just need to retrieve it.

One way of doing this is to write a custom accessor routine which knows to read from the _executor attribute and write to the executor attribute:

has executor => (
    is       => 'rw',
    reader   => '_get_executor',
    writer   => '_set_executor',
    trigger  => sub {
            my ( $self, $backend ) = @_;
            my $executor = $backend->$_can( 'run' )
                         ? $backend
                         : $self->_backend_factory( Execute => $backend );
            $self->_set__executor( $executor );
        },
    default => sub { $_[0]->_backend_factory( Execute => 'IPC::Run' ) },
);

# doesn't work in Moo 1.000007
sub executor {
    my $self = shift;
    return   @_  ? $self->_set_executor( @_ )
                 : $self->_executor;
}

has _executor => (
    is => 'rwp',
    init_arg => undef,
    handles => [ 'run' ],
    lazy => 1,
    default => sub { $_[0]->_get_executor },
);

Unfortunately the above code doesn't actually work under current (1.000007) Moo, as the custom executor accessor will be overwritten by the standard one provided by Moo (because it is created at run time while the custom one is created at compile time). This is being addressed by the Moo developers.

A workaround is possible: explicitly specify the phases during which the attribute and custom accessor are created so that the custom one is not overridden:

BEGIN {
    has executor ...
}
INIT {
    no warnings 'redefine';
    sub executor{ ... }
}

This works, but it's kludgy.

There's a also possible future problem. If (ever) Moo invokes triggers upon default value creation, there's a possibility of internal crosstalk when the run method is called if executor is not explicitly set. In that case the call stack looks like this:

proxy run()
_executor.default
executor.default
executor.trigger
_set__executor

_executor's default value constructor eventually calls its own setter. This may not be a problem. Or it might.

Two attributes are better than one, unless there are three

The above approach shows that using two attributes provides the separation required to circumvent possible infinite recursion. However, exposing either of them directly to the user require excess machinery to control their behavior.

In the next iteration both attributes are private. One attribute holds the user's backend specification, the other holds the actual backend object. The object is created on-the-fly from the specification. When the specification changes, it triggers the clearer on the attribute holding the object, so that the next time that attribute is accessed the backend object it is recreated from the specification.

To create the semblence of a single attribute the private specification attribute is initialized via the executor keyword to the constructor, and a custom executor() "accessor" was written to dispatch to the appropriate attribute for reading or writing.

has _executor_arg => (
    is       => 'rw',
    init_arg => 'executor',
    default  => sub { 'IPC::Run' },
    trigger  => sub { $_[0]->_clear_executor },
);

has _executor => (
    is      => 'rw',
    handles => ['run'],
    lazy    => 1,
    clearer => 1,
    default => sub {

        my $backend = $_[0]->_executor_arg;

        return $backend->$_can( 'run' )
               ? $backend
               : $_[0]->_backend_factory( Execute => $backend );
    },
);


sub executor {
    my $self = shift;

    $self->_executor_arg( @_ )
      if @_;

    return $self->_executor;
}

No duplicated code, no worries about tricky code paths. Not as concise as I'd have liked it. The backend object is still built upon reading or writing.

Attribute Accessor Modification

One final approach is based upon the realization that the attribute doesn't need to be coerced immediately that it is set. This can be done using a single attribute by wrapping its accessor to perform coercion at access time.

has executor => (
    is      => 'rw',
    handles  => ['run'],
    lazy    => 1,
    default => sub { $_[0]->_backend_factory( Execute => 'IPC::Run' ) },
);

around 'executor' => sub {

    my ( $orig, $self ) = ( shift, shift );
    my $attr = $orig->( $self, @_ );
    unless ( $attr->$_can( 'run' ) ) {

        $attr = $orig->( $self,
                         $self->_backend_factory( Execute => $attr ) );
    }
    return $attr;

};

The attribute default sub still has to call _backend_factory because the accessor is not invoked if the delegated run routine is called and no attribute value was specified.

The drawback with this approach that you get hit with the overhead of the coercion check with each retrieval of the attribute; with the other approaches it was only when the attribute is set.

Conclusion

In the end I settled upon using the approach with two private attributes and the custom accessor binding them. I like the simplicity of the trigger/single attribute approach, but I'm worried about code maintenance and the subtlety of ensuring there's no infinite recursion.

The implementations were more complex than I originally thought necessary for what seems to me to be fairly simple requirements.

There are a few aspects of the Moo object model that drove things in that direction.

• If the coercion routine had passed along the object, I think most of the complexity would have disappeared. While there's no guarantee that the object will be in a useable state if the coercion routine is invoked during object construction, the reference to the object is still useful as an opaque handle.

• If one could disable triggering within a trigger handler (much as in a signal handler), that would have greatly weakened the argument for attribute proxying.

• attribute setting via the default method doesn't set off the trigger handler. There may be a reason behind this behavior (or it might just have been overlooked).

• Moo doesn't seem to use attribute accessors internally (for speed I imagine), so modified or overridden ones don't always get called when the attribute is accessed.

If run-time flexibility weren't important, I think that role composition wins hands down.

I'm intrigued by the possibility of abstracting the code so that one could use either the rather more elegant role composition or method delegation.

Appendix

For completeness, here's the final implementation of _backend_factory:

use Module::Load qw[ load ];
use Module::Path qw[ module_path ];
use Module::Runtime qw[ compose_module_name ];

sub _backend_factory {

    my ( $self, $type, $req ) = ( shift, shift, shift );

    my $module = compose_module_name( "IPC::PrettyPipe::$type", $req );

    croak( "unknown $type ($req => $module)\n" )
      unless defined module_path( $module );

    load $module;

    return $module->new( pipe => $self, @_ );
}