Andreas König

This year I discussed less and got more focussed on one thing, namely PAUSE, thanks to the kisses of the muse we got flooded with, that is Kenichi, Neil, Matt, Peter and me. Thanks to that, PAUSE is, I daresay, in a better shape than ever before and definitely on a good track towards the future.

Here's a short run down on my activities at the QAH in Rugby:

Fixed a bug discovered by Leo Lapworth in rrr synchronization: it turned out that a crontab item got mangled recently.

Joined the Test2 discussion.

Set up a smoker run for 5.18.4 with the Test-Simple releases 1.302014_001..7.

Finished the pre-Hackathon activities on getting the Perl Authors Upload Server (PAUSE) to work with two separate puppet instances, one for the system operation team, and one the application (together with Robert Spier).

Added sysv startup gears for the new plack-driven PAUSE server and fine-tuning configuration (together with Slaven Rezić and Kenichi Ishigaki).

Continued the year-long activities on transforming the database of PAUSE to work without the mods table and cleaning up inconsistencies (together with Neil Bowers).

Helped out with my admin privileges on PAUSE to fix an upload glitch during Dist-Zilla-6 developer release.

Reviewed pull requests on https://github.com/andk/pause/pulls that had accumulated over the year and applied: 210, 206, 214, 216, 209, 215, 219, 207, 220, 190, 168, 172, 133, 137, 208, 212, 211, 217, and 221 (together with Kenichi Ishigaki); special thanks go to the contributors pjlsergeant, wolfsage, mohawk2, rjbs, dagolden, neilb, charsbar who generated the pull requests.

Consultation with Breno G. de Oliveira about cpanminus reporter.

Consultation with Colin Newell, Ricardo Signes, and Karen Etheridge about how to retain sane upload-permissions for collaborating groups of users.

Consultation with Joel Berger on a potential Mojolicious cookbook article about when and when not to use weaken.

Thank you to the sponsors, FastMail, ActiveState, ZipRecruiter, Strato, SureVoIP, CV-Library, OpusVL, thinkproject!, MongoDB, Infinity, Dreamhost, Campus Explorer, Perl 6, Perl Careers, Evozon, Booking, Eligo, Oetiker+Partner, CAPSiDE, Perl Services, Procura, Constructor.io, Robbie Bow, Ron Savage, Charlie Gonzalez, Justin Cook, many of us would not have been able to come and stay for four days without their help. Thanks to Neil and Barbie and JJ for setting all this up and managing everything so nicely and smoothly. Thanks to Wendy for keeping us sugared and vitamined and fed and hugged and hugged and hugged.

It was awesome. Thanks so much to all who made this possible: Laurent Boivin, Book, Wendy, all the sponsors (http://act.qa-hackathon.org/qa2014/sponsors.html : Booking.com, Splio, Perl6, ELIGO, Grant Street Group, Dyn, EVOZON, Campus Explorer, elasticsearch, Mongueurs de Perl), and all participants: you were great, you are great.

As has been noted, the value of the get together often is the groundwork for lifting new ideas off the ground. And so it was for me, too:

Talk with Christian Walde, Slaven Rezic and David …

I don't think this is documented anywhere, so let me blog about it before I forget it. Install the module CPANDB. See if it works by running one sample program, maybe:

  use CPANDB;
  use YAML::Syck;
  print Dump [CPANDB::Distribution->select
             ("order by volatility desc limit 1")];

It may take a while but you will get something like this:

!!perl/array:CPANDB::Distribution 
Exporter
'5.63'
FERREIRA
1
perl
FERREIRA/Exporter-5.63.tar.gz
[...]

The kick now is that after such a command you have the current database on your local disk. I found it in my homedirectory under

.perl/ORLite-Mirror/CPANDB.sqlite

Now you can access it directly and find, e.g., the modules depending on Catalyst::Devel with:

  % sqlite3 ~/.perl/ORLite-Mirror/CPANDB.sqlite
  SQLite version 3.7.4
  Enter ".help" for instructions
  Enter SQL statements terminated with a ";"
  sqlite> select distribution from requires where module='Catalyst::Devel';
  Catalyst-Authentication-Credential-OpenID
  Catalyst-Example-InstantCRUD
  Catalyst-Example-InstantCRUDStylish
  Catalyst-Helper-AuthDBIC
  Catalyst-Helper-View-TT-Bootstrap
  Catalyst-Model-Cronolog
  Catalyst-Model-Estraier
  Catalyst-Model-JabberRPC
  Catalyst-Model-SVN
  Catalyst-Model-XMLRPC
  CatalystX-Starter
  Handel
  Task-BeLike-MELO
  Task-CatInABox
  Task-Catalyst
  Task-Catalyst-Tutorial

And what is most amazing about it? That 186 or 46% are either resolved or patched. That's a wonderful success rate.

If you have a login on rt.cpan.org you can issue this query to see the actual 401 tickets.

Very nice graphics and ad-hoc statistics are then provided by RT when you look at the bottom of the page. I've never noticed that RT footer under the result of a query before.

For example I can create a bar chart by CreatedMonthly that shows me when the rally started: 65% of the tickets were created from August to December. That's because on the first days of August was the relaunch of http://analysis.cpantesters.org/ after a few months break due to the cpantesters 2.0 launch. Analysis is the real work horse that drives me in this game. It's kind of playing solitaire with bugs. Analysis provides the deck: cards of many different shapes and colors, my job is to read them and find the ones that can be resolved quickly.

A never ending deck it seems. In the last four years we had over 3400 uploads to CPAN that have both passing and failing tests. Enough material for more players. Feel yourself invited.

The total number of my posts to rt.cpan.org over the years approaches 1500 tickets. 840 of them are resolved. That's a lot and then again it's a lot that remains unresolved. The first thing you have to learn if you're about to follow my footsteps: don't expect too much and be patient. Today I can enjoy mails about resolved tickets I have opened years ago and hardly can remember. They are the final intrinsic reward for the hours spent in the RT ticket game.

Analysis has changed the game drastically though. In the old times it was a purely random process when I was hitting a distribution with a failing test. Only when some other channels brought my attention to a distro and a test of that distro failed then I had a case for a potential bug report. Analysis now brings the distros to my (and your;) attention nearly instantly. As soon as it has at least three PASS and three FAIL reports it gets listed in analysis. This may be on the same day as the release. I usually wait until three whole days have passed before I allow myself to report. I think this is a polite grace period that gives the authors a break.

If a bug report arrives as quickly as 72 hours after the release many authors are still familiar with the process of releasing and immediately make a new release. That's so fascinating. Getting 46% of the tickets resolved within a few months was not a goal you hoped to reach in the old days. The standard seems to be shifting. Thank you, all.

PS: if you know of other software QA projects using multivariate statistics to identify potential causal relations, please post them here. I'm eager to learn about other approaches.

Every so often you may have found yourself puzzled by the results the cpantesters provided, wondering which reasons may stand behind the fails or passes.

So you click on one of the FAIL reports and it doesn't reveal anything useful to you?

Chances are that if the reports contain some common pattern the new analysis site will point it out to you. Analysis uses some well known basic statistical tools that estimate the influence of independent variables on the result of the test runs.

During the last two months I wrote over two hundred RT tickets based on the findings (see also my use.perl.org posting). Among my first favorites were FAILs that happen only when uselongdouble is defined, like #51196. There are plenty of similar ones. Another case where the tables provided a guaranteed quick understanding of the problem was when an upgraded prerequisite broke the distro, like #52889. See how the table at analysis marks all Test::Builder versions above 0.84 as red. You can skip reading reports and go directly to the Changes file of Test::Builder to see what they changed around 0.84.

What I'd love to see is that cpantesters got some inspiration from reading these tables. When I see a distribution with only a dozen test results and plenty of strong correlations I try to create reports that add new flavors to the existing set. After a few hours or days I visit the cases again and sometimes the number of strong correlations has dropped and a clearer picture evolves where the hot spot is.

Another use of the site is to identify broken cpantesters setups.

Another one to discover bugs in bleadperl, as in #50939.

I'm sure you'll find your own entertaining bits in it. Enjoy!