On prototyping in public ... the long version
I recently posted a quick update on the p5-mop project the other day, something I have been meaning to do for a long time. I am sure given the slow and often rocky progress of this project that many people have their doubts if will ever see the light of day, and to be quite honest, some days I found myself doubting it as well.
As I mentioned in my previous post, I am going to be changing jobs and moving my entire family over to the Netherlands, this will no doubt occupy just about all of my time for the next several months, meaning that any progress will be slow. The planning for this move has been in the works for a while now, which is a large part of why progress over the last several months has also been quite slow. But I actually think that ultimately this has been a good thing for the project, because it has forced me to step back, learn, think and design; rather then dive in and just write code.
I personally have high hopes for my latest approach mostly because I am now (finally) really diving into the guts of perl and trying to write something at that level, rather then trying to add something new on top. In hindsight, I believe that approach was a large part of the reason why the first prototype ultimately failed and why I am currently shelving the second prototype.
I designed the original p5-mop to be a totally independent object system that strove to not mix with the existing core. As I said, the design failed because I had spent too much time focusing on keeping the p5-mop object system separated, that I failed to notice some serious show stopping flaws in the ways in which it ultimately did have to connect to the perl guts (this was all explained here for those interested in the gritty details). This outcome was disappointing to say the least, all the effort I (and others) had put into designing the syntax and semantics felt as if they were for naught. Some people tried to encourage me to take a step back and try a simpler approach, but as my family will tell you, I can be very stubborn; so I instead decided to go off the deep end and propose a re-write of the core.
The Moe project started off with a bang, but as I soon quickly realized Perl (the language) was not Perl without the CPAN and the CPAN was not possible without perl (the interpreter), or more specifically XS. My announcement, in the end, turned out to be extremely premature as I hadn't yet done the analysis of CPAN to see what would need to be written (and re-written) in order to make Moe a viable replacement for Perl (spoiler: pretty much everything on CPAN). After doing that, and realizing that it ultimately would not work, I re-dubbed Moe "a thought experiment" and proceeded to work on it without care of any backwards compatibility concerns. While this project failed its original goal of replacing perl, it succeeded in making me realize that my original "keep them separated" p5-mop approach was all wrong and gave me a much deeper appreciation of the perl core. Then while watching Pete Martini's excellent talk at YAPC::NA something clicked in my brain! I realized that I needed to work much more "inside the box", meaning I needed to approach p5-mop with a better understanding of (and a deeper respect for) how perl (the interpreter) actually worked.
On the plane home from YAPC::NA that year I started to sketch out an idea for a different approach in which I attempted to stay true to the perl internals (as I understood them at the time). This became the p5-mop-redux project which I worked on feverishly for several months refining the original vision more and more. In this project, I made sure to learn from one of the core mistakes of its predecessor; very early on in the project I ported a decent sized perl module. The process of doing this allowed me to shake out a number of bugs and work out details for key elements of the design. After a few months we actually released a version of this to CPAN and encouraged people to try it out (and many actually did). Before the second release, my co-conspirators, Jesse Luehrs and Florian Ragwitz, removed all but one dependency and re-wrote a significant portion of it in XS, which we knew was a necessary step. Then came the holidays and other $work based distractions and the project stalled as myself and the other contributors simply got too busy.
During that busy time I had very little time to program, but I had a lot of time to think and learn. I realized that I really needed to wrap my head around XS and C otherwise my further contributions would be severely limited. I began by pulling out my dusty old copy of the K&R book, and progressed through that into perlxstut, perlxs, perlguts and finally the perlapi docs themselves. I even ordered a used copy of "Extending and Embedding Perl" and pretty much read it cover to cover. After several months, I felt that I was finally ready to jump in and pick up the p5-mop-redux project from where we left off. Then after spending a few days staring at the design I realized that, once again, I had built something which failed to integrate well with the perl core. The assumptions I had made in the original design seemed perfectly reasonable when viewed from the perspective of a Perl programmer, but when viewed from my new perch atop the perl guts, it had some serious flaws.
All this is not to say that the two prototypes and the Moe experiment were a waste, quite the contrary, this journey has been a very humbling and extremely interesting learning experience for me. When I first set out on the goal of designing a new object system for Perl, my own ignorance (and arrogance) lead me to believe it was best to stay separate from the core. Perl, the immovable rock of backwards compatibility that it is, would ultimately not allow me to do this until I showed the proper respect and humility. And so was born p5-mop-XS, which I hope will be the final prototype if for no other reason then the fact that hopefully "the third time is the charm".
I cannot resist saying “I told you so” about Moe. :-)
The thing is, no matter badly the backward compatibility constraint sucks, you absolutely need it to succeed. If you remove it, you basically become Perl 6 – not in how the design is going to end up, but in scope and ambition. And we already have one Perl 6 – it’s unlikely that another will succeed. If you try to be less ambitious then, it’s almost certainly a death knell: you are unlikely to deliver enough incentive for anyone to switch from the old to the new. And old has a massive userbase advantage to make up for its flaws in competing against you. Look how that worked out for Python 3 e.g.… “not Perl 6” is not looking like such a hot idea any more, does it? (Admittedly their list of mistakes is longer than just this one factor. But this one had the potential to greatly diminish any number of other mistakes.)
That’s not to say the backcompat constraint is fun… but it’s necessary. “Immovable rock” is not a bad way to put it.
I’m also struck by how much this whole journey mirrors the programmer’s usual reaction to trying to read someone else’s code: “this sucks, I’ll just write it from scratch myself.” :-)
Anyway, I’m anxious to see where you take it from here.
Yeah, I kind of knew Moe was never going to succeed in the end, but it was a fun project to hack on. And like I said, I learned a whole lot by doing it and it did seem to help push some of the other "re-write perl" projects to start working too. In my talk at YAPC::NA Austin I said that I think this kind of experimentation is important for the future of Perl 5, because if people bring good ideas back to the core Perl 5 codebase then ultimately we all will benefit.
You have looked at the guts of Perl 5 right?
Unless you are a full-on Unix neckbeard of the highest order then the only sensible reaction is to not want to wade into that swamp.
I advocated your project last weekend at Polish Perl Workshop in Poznań. It was just a lighning talk about mop and extending it's traits: http://bit.ly/1jFqKfj
Oh I won’t call it well written, believe me. :-) I’ve tried to patch it at times, succeeded with trivial stuff outside the interpreter guts, and I’ve read others’ patches. I don’t have a firm grasp of it but I do have a sense of what it’s like to work with. It’s not as terrible as its reputation, and there are some really clever bits (that I mostly wish weren’t necessary…), but it’s no kind of beauty. Kinda like Perl-the-language (but with its coherence more corrupted by time), come to think of it.