Stevan Little

I recently posted a quick update on the p5-mop project the other day, something I have been meaning to do for a long time. I am sure given the slow and often rocky progress of this project that many people have their doubts if will ever see the light of day, and to be quite honest, some days I found myself doubting it as well.

As I mentioned in my previous post, I am going to be changing jobs and moving my entire family over to the Netherlands, this will no doubt occupy just about all of my time for the next several months, meaning that any progress will be slow. The planning for this move has been in the works for a while now, which is a large part of why progress over the last several months has also been quite slow. But I actually think that ultimately this has been a good thing for the project, because it has forced me to step back, learn, think and design; rather then dive in and just write code.

I personally have high hopes for my latest approach mostly because I am now (finally) really diving into the guts of perl and trying to write something at that level, rather then trying to add something new on top. In hindsight, I believe that approach was a large part of the reason why the first prototype ultimately failed and why I am currently shelving the second prototype.

I designed the original p5-mop to be a totally independent object system that strove to not mix with the existing core. As I said, the design failed because I had spent too much time focusing on keeping the p5-mop object system separated, that I failed to notice some serious show stopping flaws in the ways in which it ultimately did have to connect to the perl guts (this was all explained here for those interested in the gritty details). This outcome was disappointing to say the least, all the effort I (and others) had put into designing the syntax and semantics felt as if they were for naught. Some people tried to encourage me to take a step back and try a simpler approach, but as my family will tell you, I can be very stubborn; so I instead decided to go off the deep end and propose a re-write of the core.

The Moe project started off with a bang, but as I soon quickly realized Perl (the language) was not Perl without the CPAN and the CPAN was not possible without perl (the interpreter), or more specifically XS. My announcement, in the end, turned out to be extremely premature as I hadn't yet done the analysis of CPAN to see what would need to be written (and re-written) in order to make Moe a viable replacement for Perl (spoiler: pretty much everything on CPAN). After doing that, and realizing that it ultimately would not work, I re-dubbed Moe "a thought experiment" and proceeded to work on it without care of any backwards compatibility concerns. While this project failed its original goal of replacing perl, it succeeded in making me realize that my original "keep them separated" p5-mop approach was all wrong and gave me a much deeper appreciation of the perl core. Then while watching Pete Martini's excellent talk at YAPC::NA something clicked in my brain! I realized that I needed to work much more "inside the box", meaning I needed to approach p5-mop with a better understanding of (and a deeper respect for) how perl (the interpreter) actually worked.

On the plane home from YAPC::NA that year I started to sketch out an idea for a different approach in which I attempted to stay true to the perl internals (as I understood them at the time). This became the p5-mop-redux project which I worked on feverishly for several months refining the original vision more and more. In this project, I made sure to learn from one of the core mistakes of its predecessor; very early on in the project I ported a decent sized perl module. The process of doing this allowed me to shake out a number of bugs and work out details for key elements of the design. After a few months we actually released a version of this to CPAN and encouraged people to try it out (and many actually did). Before the second release, my co-conspirators, Jesse Luehrs and Florian Ragwitz, removed all but one dependency and re-wrote a significant portion of it in XS, which we knew was a necessary step. Then came the holidays and other $work based distractions and the project stalled as myself and the other contributors simply got too busy.

During that busy time I had very little time to program, but I had a lot of time to think and learn. I realized that I really needed to wrap my head around XS and C otherwise my further contributions would be severely limited. I began by pulling out my dusty old copy of the K&R book, and progressed through that into perlxstut, perlxs, perlguts and finally the perlapi docs themselves. I even ordered a used copy of "Extending and Embedding Perl" and pretty much read it cover to cover. After several months, I felt that I was finally ready to jump in and pick up the p5-mop-redux project from where we left off. Then after spending a few days staring at the design I realized that, once again, I had built something which failed to integrate well with the perl core. The assumptions I had made in the original design seemed perfectly reasonable when viewed from the perspective of a Perl programmer, but when viewed from my new perch atop the perl guts, it had some serious flaws.

All this is not to say that the two prototypes and the Moe experiment were a waste, quite the contrary, this journey has been a very humbling and extremely interesting learning experience for me. When I first set out on the goal of designing a new object system for Perl, my own ignorance (and arrogance) lead me to believe it was best to stay separate from the core. Perl, the immovable rock of backwards compatibility that it is, would ultimately not allow me to do this until I showed the proper respect and humility. And so was born p5-mop-XS, which I hope will be the final prototype if for no other reason then the fact that hopefully "the third time is the charm".