Why I Stuck With Perl

I've just read a very thoughtful description of why one Perl programmer switched to Python. In this blog post, I'll explain why I did not.

When I started on the Marpa project, Perl was not an automatic choice. In fact it looked like it might be the wrong one. There was as much buzz for Python then as now. Maybe more. I'd had my own experience with Python, and it was excellent. I'd used Python to create a threaded mail client for testing purposes. I found Python to be easy to learn, fun to use, and an effective way to accomplish my task.

Perl had another problem. An important audience for Marpa, my new parser, is academia. Academics certainly use Perl. My experience is that they even respect it in a certain sense. But Perl does not fit neatly into the minimalist paradigms which still rule academia. If you speak Perl to academics, you are not talking their language. That means you run a great risk that nobody will listen.

I read and studied a lot about the relative merits and demerits of various scripting languages. Others have covered, and continue to cover, that ground very well. Let me go straight to explaining why I stuck. Perl has CPAN. And CPAN ... well, let me tell you a story.

An Incident on CPAN

Recently, my development version of Marpa::XS showed a lot of failures from the cpantesters. This happened immediately after a mild brag in this blog, about my impressive success rate in writing portable C code. It was like one of those Greek myths, where one of the Gods in Olympus decides a big-talking mortal needs to be taken down a peg.

I did not receive any lightning bolts, but I sure got a variety of test failures. With considerable effort, I was able to eliminate all but one of these on my own. The remaining failure happened only on the tests run by Andreas Koenig. Not that I expected Andreas's setup was at fault -- it was more likely that all the other setups were being liberal where Andreas was being strict. Which is what turned out to be the case.

So I emailed Andreas. Frankly, I did not have high expectations. Andreas is doing a lot of important things. My guess was that discovering why new C code on the unstable development version of a new module does not execute in his test setup might take quite a while to percolate to the top of his "to do" list.

I was therefore very surprised when the answer rocketed back. My development version of Marpa::XS failed only with the development versions of Module::Build later than a certain version number. Andreas, it turns out, is set up to do regression analysis, and the regression analysis pointed straight to the combination.

For those of you who don't know, regression analysis essentially sets up an equation combining various factors (in this case, module versions, Perl versions, hardware, OS, etc.) with an outcome (in this case, success or failure). It can be used to see which factors correlate with which outcomes -- in this case, which combinations of modules, hardware, etc. are correlated with failures.

It's a very cool technique. (I tried to get my clients to use it in my consulting practice, but there was just too much pushback on the math.) Andreas's regression tools pointed straight to the problem. It began to happen when Module::Build switched to non-lazy loading of dynamic code.

Here's what my bug was. I had declared a C function, call it xyz(). But I had never defined xyz() -- there was no code for it. But I also never called xyz() anywhere. With lazy dynamic loading, that meant no attempt to load xyz() was ever made. So with lazy loading, my bug was harmless.

In the name of thorough testing, Module::Build was switching to non-lazy loading. This meant that, call or no call, my executable looked for xyz() and, when it did not find it, abended.

In his email to me, David Golden left it an open question whether testing with lazy or non-lazy loading is best. I'll leave the final resolution of that to his capable judgement. For Marpa, I think Module::Build's new behavior is the best, and I've changed my test setup to always be non-lazy. In the future, any never-called never-defined function like xyz() will abend in my test setup.

Summary

My quest is to make Marpa's C-enabled version widely portable. To this end, CPAN not only offers me free widespread testing, but even fast, polite, high-quality help with debugging. Would I get that if I were programming in a language other than Perl?

Note 1: Marpa parses any grammar that you can write in BNF, and parses all grammars parseable by yacc or recursive descent in linear time.

15 Comments

This is awesome. In light of the experience that some have had with Perl guru's being less than overly helpful, it's good to see that the community in general is still awesome. If it were not for CPAN I don't think I'd be writing perl today. This is just further evidence that trusting in CPAN is absolutely worthwhile. I like writing code and I hope to never end up as an example on The Daily WTF. I trust CPAN to help me realize that hope :) Thanks for your efforts.

Is it really that the "gurus" (an overused word) are less than helpful? Maybe you just came across the wrong ones?

Most of the very core of the community are very approachable and infinitely helpful. In this case, Andreas, is a particularly good example. He's one of the all around nicest guys I have interacted with in [fF]ree software context. Curiously, the other name mentioned, David Golden, rings a similar bell. Thought- and helpful, always polite and incredibly industrious.

Off the top of my head, there are at least 20 other people I could name who are in the same league as the above two. I could come up with a few more who have edges but are generally nice and helpful once you see past the first impression.

In my experience, there are very few members of the community who actually matter but who are not approachable. This is largely because they wouldn't be part of the community otherwise.

I wish more people register the regression testing project - there is nothing else like it! By the way here is Andreas blog post about it: https://blogs.perl.org/users/andreas_konig/2011/01/guess-who-has-written-401-rt-tickets-in-2010.html , apparently the regression data is available at: http://analysis.cpantesters.org/ for everyone to browse.

The anecdote doesn't really illustrate why you chose Perl over Python, apart from making an inference that Perl's community is "nicer" than all other languages.

I agree with you that Python is both easy to learn and fun to use. I program in both languages and I believe Python is easier to learn (its syntax being my primary reason) and I think Python is more "fun" -- if only because there seems to be a wave of new programmers who have discovered the joy of hacking (again largely in part to the simpler syntax) -- and there seems to be a better hacker culture.

If anything, I've found my learning experience with Python to be easier (and thus more enjoyable) than with Perl. As the linked article points out, the only downside with Perl seems to be the stagnation, which does not gel with the hacker ethos.

@stewheck: stagnation isn't really a problem with Perl any longer. 4 years ago, maybe, but now here's tons of work being done. New features being added to each release, new features being proposed and discussed, ctypes work, new web frameworks, and a growing conference community are just a few points. Hell, Blogs.perl and ironman alone show show the growing buzz. It's only a matter of time before that buzz starts to overflow beyond our community borders.

I use Perl because it's awesome. I don't really need any other reason besides that. It's pretty, unbelievably flexible, and has has a bright future. It's even more hip than my new Seven for all Mankind jeans, but without half of the pretentiousness, which, if I may say, is a fairly difficult feat.

> The anecdote doesn't really illustrate why you chose Perl over Python

Did you read the summary? I think it's fairly clear that he's praising the testing infrastructure Perl has that allows you to get a distribution tested across 7 OSes and 5 major (plus a rat's tail of minor) Perl versions [ http://matrix.cpantesters.org/?dist=Marpa+0.202000 ] within a few days of upload and then to get automated analysis of one's failure reports to pinpoint the probable causes [ http://analysis.cpantesters.org/solved?distv=Plack-App-Proxy-Selective-0.08 ].

And, the crux: All this without paying a single cent.

It helps to realize just how big and valuable this is when you start to think about what it would flat out cost you if you were to try and replicate that on your own.

Also, the linked article is written by someone who simply put, does not know Perl 5 very well and mainly talks about Perl 6. So please take anything written in there with a big pinch of salt.

Is "taking advantage of an enormous and useful ecosystem to improve the quality and efficacy of my software" not reason enough to choose Perl over Python? That's the central point of the anecdote.

Andreas does not get anywhere close to enough accolades for all his work.

Perl _is_ "awesome", but I'm hoping that Perl6 arrives sometime before your cool jeans are out of fashion ;) Python is definitely in vogue at the moment -- all the cool kids are doing it.

In short, no. Perl might have a bigger ecosystem, but I just think the rate of change for Python is greater. The headline is somewhat misleading as there is less of a comparison being made here, and more of a story about how good Perl's ecosystem is, i.e. not much mention of Python's ecosystem. And I don't think the size of an ecosystem is reason alone. The biggest factor is uptake and evolution. Sure, good things come to those who wait, but after using Perl5 for years I got tired of waiting. btw, hold back the flamethrower chromatic, I've actually taken part in the Perl ecosystem myself (sent you some errata for Perl.com articles you've written, conversed with you about Perl stuff ages ago via Twitter/email). I'm not part of the Perl aristocracy, but then I do think there is a bit more attitude on that front with the more senior Perl programmers I know (not you). Python might be Perl's simple cousin, but he's actually pretty quick on the uptake ;)

Yes, I read the summary. I'll tell you a story about apples. I'll go into detail about the benefits of apples and why they taste so good. Then I'll briefly mention that I had a sour orange once. Not only is it apples vs oranges, there wasn't really a comparison. I'm not bagging Perl, can the Perl zealots please calm down. I used to be one of you :P

The size of the ecosystem is less important, to me, than the existence and quality and coverage and agility of CPAN Testers.

Can you see how I consider it an advantage over every other language I've used?

Didn't you first write Marpa in TCL?

About Jeffrey Kegler

user-pic I blog about Perl, with a focus on parsing and Marpa, my parsing algorithm based on Jay Earley's.