A funny thing happened on the way to the p5-mop...

So, I mentioned in my recent talk that the p5-mop project is currently stalled but did not really expand on that topic much. In this blog post, I'm going to explain what has happened over the past year or so with the project, discuss some issues we ran into while trying to implement the p5-mop, and lay out why I think we need to do some work in directions other than the p5-mop for a while in order to get back the p5-mop back on track.

First let me start out saying that p5-mop is not dead, and that I would very much still like to finish implementing the "on CPAN" version of it. I believe Jesse "doy" Luehrs, my co-conspirator, feels this way as well. My outlook on taking that "on CPAN" version and merging it into the p5 core is currently somewhat less optimistic for reasons I will explain.

The Current Problems

The first problem with the p5-mop project is that, like all deep magic around extending the p5 core via modules, it is very mentally taxing and it takes a good day or two to re-wrap ones brain around it after leaving it, even for only a few days. This makes it very tricky to work on while still doing paying work at the same time (this is true of any really ambitious open source project, so this should come as no surpise). So while it is fortunate for Jesse and myself that our work lives have been busy lately, this does have the unfortunate side effect of neither Jesse nor I being able to pay enough attention to the p5-mop project.

That first problem is probably surmountable, but that is not the only one we have encountered. The second problem is actually just getting an implementation to work -- sanely -- in a "on CPAN" version, on top of Perl 5. Our original prototype used a lot of PadWalker and Scope::Guard magic to make attribute access work just like lexical variable access. Unfortunatly, our attempts stressed PadWalker and Scope::Guard, causing them to break in fabulously strange ways.

Our intention had always been to replace the PadWalker/Scope::Guard bits from the prototype with XS equivalents down the road, so when we ran into this problem initially, we tried to work around it by advancing the XS version instead. This ended up being pretty complex to actually implement, and is currently unfinished.

The prototype also depends on Package::Anon to handle method dispatching. This was also just a temporary hack in service of getting the prototype up and working. Unfortunately, in trying to make Package::Anon work and experimenting with how it "should" work, we ran into several limitations and bad assumptions within Perl's method dispatch. In particular the assumption that a class is a string which must be the name of a package, was very problematic. In the end, we were forced to conclude that Perl's method dispatch mechanism would almost certainly need to be changed to allow some kind of non-package based dispatching. This probably means some core level work, but we stopped short of digging into this deep enough to say for sure.

The Future Problems

Those are the two biggest issues we ran into (so far, anyway), but they are very complex and involved problems. Either one alone would be a significant hurdle, but they actually interrelate in fairly fundamental ways, making the overall complexity level of fixing them incredibly high. I do believe, given enough tuits, that they can be solved and a solid working prototype could be put on CPAN for people to start working with. But once on CPAN, we must ask the question, "What next"?

This is where I began pondering the Perl 5 core and what dropping a complex MOP into it might mean for the maintainers and even the codebase itself. One of the main goals behind adding p5-mop to the core was to try to unify the many different object implementations out there, but honestly it has been very difficult for both Jesse and myself to see a way forward that doesn't end up with a situation that is even more fragmented than what we have now.

These thoughts, along with a small detour into the sad state of asyncronous programming in Perl and a little bit of reading of the recent p5p "sub signatures" thread (and other related threads), really got me thinking a lot. It was all this which lead me to write my talk for OPW 2013. Not until I actually got to writing the slide about p5-mop did I realize that yes, it was stalled and that I did not currently know how to move it forward.

So, where does this leave us now?

In his talk about the future of Perl 5, Jesse Vincent laid out a plan for cleaning up the Perl 5 core in such a way as to maintain our backwards compatability commitments, but still be able to move the language forward. My experience with trying to implement the p5-mop on top of Perl 5 has now convinced me that achieving this goal of a cleaner, simpler core is critical for any sort of significant continued innovation or evolution of the Perl 5 languange.

This is why I am currently directing my efforts towards a prototype of what I envision an "Ultra Modern" Perl might look like. This will include a MOP very much like the p5-mop along with some other features I spoke about last year at DCBPW.

It will also (and perhaps more importantly) try to actually spec out what a "smaller p5 core" might look like from a language perspective. It is my belief that this admittedlly odd digression from the p5-mop project is actually neccessary to ensure the success of the project overall.

Am I right? Am I crazy? I guess only time will tell. In the meantime, if you'd like to help us actually try and figure out what an Ultra Modern Perl looks like, the MoeOrganization is over here and everyone is welcome to join.

(NOTE: Thanks to John "genehack" Anderson for his editing help on this article and much better title suggestion).


Just for the record. I believe that your (plural) work on p5-mop is immensely valuable to the community regardless. I feel greatly indebted to you for tackling the hardest problems so I don't have to.

Thanks for explaining it.

I wonder if p5-mop could have been (could yet be?) more successful if it wasn't so ambitious.

Yes, having attribute access just like lexical variable access would be cool, but I could certainly live without it.

We could put Class::MOP in core now (if enough people were in favor).

How was p5-mop going to improve on that?

What would an 80% solution p5-mop be like?

I'm concerned that there's this constant goal of /must work without changes to the core/. Would this be easier if you made changes to the core? could those same changes be useful for other things if building this didn't pan out?

Firstly, thanx for being frank about where you stand. It's wonderful to see that you feel comfortable expressing these issues to all, not just to the inner circle.

So, you've concluded you must re-design core in order to advance Perl? I say go for it.

It's basically a Follow-your-dream scenario.

Jeffrey Kegler does it with Marpa. I do it likewise (living off savings) to do what I want.

I refuse to believe that the only way we can contribute is by working a 9..5 job.

Forcing ourselves to fit that role is the same as forcing all our future programs to fit the one-and-only-core of Perl. For some of us, that's not going to happen.


In the end, we were forced to conclude that Perl's method dispatch mechanism would almost certainly need to be changed to allow some kind of non-package based dispatching. This probably means some core level work, but we stopped short of digging into this deep enough to say for sure.

I think a lot of these issues boil down to "Core infrastructure doesn't enable us to do these magical things yet". That doesn't mean it's impossible, but it sure makes it harder. I think we should add that infrastructure first.

Stevan, I sort of assumed that the point of a "core" MOP is to get away from blessed references and that any CPAN implementation would be a crude prototype of such.

On the topic of attributes, I meant that having them behave *like* lexicals was the thing I could live without. If I had to use an accessor to get to an attribute, so be it.

I was hoping the MOP work would define semantics for an alternative to blessed references, that it would go into the core as a new type of SV, and that given *that*, then new mechanisms for method dispatch could be written.

That would let blessed reference objects work as they do and let new MOP objects coexist (hopefully better).

Easy to say, I know, and probably hard to code, but I think it could be a way to sidestep these problems.

Stevan, about forking Perl:
Why Scala? Couldn't be just plain old Java?
Using Java will (IMHO) dramatically increase the number of people willing to play/help with it and at least number of watchers on github...

We (even plain users like me) could just look at how, for example, some things are implemented for Groovy and build on that.
So at least why not lower the entry barrier to this project.
Also it will be closer to the JVM.
Business people may also be more interested.

Krasimir, the "Why Scala?" question (and in particular, "Why not Java?") is addressed in the moe FAQ.

Why Scala?

Leave a comment

About Stevan Little

user-pic I blog about Perl.