A late Perl QA Hackathon 2012 post scriptum
First post, the post that hurts the most. Well ... not ... but still in time before the y2k12-maya-bug triggers.
I am late in writing this summary about my hackathon but it fits the prolongation style I exercised this time. I had quite a slow start as I found it difficult to flush my overfull @work mindset and resume my open source projects. I used my flight delay to carefully prepare a TODO list which finally helped on that flush'n'resume exercise.
So what did I do?
My pet project is benchmarking Perl. There I have one major problem:
- The visible benchmarks are boring and the interesting parts are invisible.
My benchmarks are rather straight lines without interesting changes: they are straight in the 5.8 timeline; they are straight in the 5.10+ timelines.
However, both straight lines are different to each other, so obviously something must have happened during the 5.9.x times. Unfortunately, exactly that interesting timeline did not work well in my benchmarking toolchain.
So the mission was:
- find out the reason and fix it
- then kind of semi-bisect into interesting commits in 5.9.x
- then proceed with more micro benchmark to elaborate what happened during 5.9.x
So the high-level plan for my hackathon attendance was kind of a big polishing Chuck Norris roundhouse kick. This is a bit in contrast to what I read so far from others about inventing amazing new stuff - anyway, I think someone has to solve the boring parts of benchmarking.
Results:
- organized backup of my infrastructure and results DB
- reworked reporting of codespeed information (codespeed is the graph rendering app)
- rework details on the reuse of already built Perls and CPAN dependencies
- review/rework/drop troublesome benchmarks (crypt_primes, P6STD, MooseTS, RegexpCommonTS)
- refreshed my collection of 100+ Perls to apply all those changs
I did not yet come to the point of actually running and inventing new benchmarks with 5.9.x, but it's now possible.
On several other "second fronts" during Perl compilation and benchmarking I did:
- play with the open64 compiler
- extend dpath to be a better companion in unixoid oneliners
- learned how to create Debian packages from CPAN modules
- migrate to Dist::Zilla (pluginbundles plus some distros)
- find and carve out yet another INI module that works better for deeply nested data
As slow as the hackathon started - it return it kept its momentum for me during the next couple of weeks to happily continue these projects in my rare spare time.
Thanks to the organizers! Everything felt easy and natural which is always only achieved by hard preparation work.
Thanks to the sponsors.
Born. Became programmer. Atari Basic, Pascal, C, C++, Java, Perl. In this order. :-) Played International Karate, Super Cobra, Loom, MechWarrior 1-4 and Quake 3. Studied computer science, Debian, Emacs, Perl. Active member of Dresden Perl Mongers. 
Well, I should have added at least soome URLs into the Benchmarking. Here we go:
Overview and results.
I was wondering why someone cares about 5.8 and 5.10 now that we are heading to 5.16 but then I
click through the "results" link added in the comment.
The graph there shows - if I understand it - that starting with 5.10 (an including 5.14) perl is 20% slower than 5.8.
What I can also see is that since April the 5.15 branch got the 20% back. What happened? Have you found the issue and was it fixed?
Some answers in reverse order:
The first rule of the benchmark club: you do not change the environment of the benchmark.
In other words: my changes during the hackathon could have influenced the measurements. I don't think they did, but in fact this happens more often than one might hope. For instance, I also upgraded my CPAN mirror and dependecies during hackathon. Only rerunning the old versions again will make sure. I planned to do that anyway.
"The issue" that happened after 5.8 is not yet clear, that's what I try to narrow down by running benchmarks against 5.9.x. Most probably it will not be a single issue but somewhere in the generalisations for better flexibility that made 5.10 the base for other nice developments that happened after that.
Another theory: I'm mostly running threaded Perls so far. It could just be a change in the overhead against non-threaded Perl that got worse and now becomes better again.
Re on caring for 5.8. It is natural because it is that famous Perl that lots of people used for many years and which still CPAN authors respect when writing libraries. It is therefore the speedy baseline worth to compare against.
From the performance perspective, according to Reini, it's not even the fastest Perl, but I can't get my toolchain to run with 5.6.