July 2013 Archives

Smoking CPAN in one line

I'm using CPAN::SQLite. I don't like it and I'm trying to improve the locking issues with concurrent process all trying to read from cpandb.sql by writing a new CPAN::UnQLite. But this is something else.

Today I want to smoke whole CPAN in one line to check the new experimental return-or check by Niels Thykier for "Possible precedence issue with control flow", for which we filed several bug reports in the last two days. See https://paste.debian.net/16932/ and perl #59802. To repro add this patch from Niels to latest blead, if it's not already included.

$ cd ~/.cpan
$ alias p='perl5.19.2d-nt@1292fdb2'; alias ppan='AUTOMATED_TESTING=1 PERL_MM_USE_DEFAULT=1 PERL_EXTUTILS_AUTOINSTALL=--defaultdeps p -S cpan'
$ cat allmods.sh
sqlite3 cpandb.sql 'select mod_name from mods group by dist_id order by mod_name;'

$ for m in `./allmods.sh`; do echo $m; ppan -t $m 2>&1 | tee -a log.cpan | grep "Possible precedence issue with control"; done

And in a 2nd term watch it flow, and react on keyboard input.

$ tail -f log.cpan

Note this is just a oneliner, not a real smoker, which needs to setup some environment variables and disable STDIN. For smokers see http://wiki.cpantesters.org/wiki/QuickStart

Addendum, keep some dependencies

cpan -t does only testing, but not installing. Major dependencies get built and tested all over again, if they are not installed. This costs some unnecessary minutes. So I added this in a seperate term to keep the most prominent deps.

$ alias jpan='ppan -j ~/.cpan/CPAN/nosqlite.pm'
$ jpan `grep -a "Warning: prerequisite" log.cpan|cut -d' ' -f3|sort |uniq -c|sort -rn|perl -alne'print $F[1] if $F[0] > 5'`'

I need to use a version without use_sqlite to avoid the CPAN::SQLite locking bug. Yes, I've tried to fix it, but some writing codepath still holds a reference to a stmt handle which prevents the readers to be shared.

cd ~/.cpan/CPAN/
sed -e"s,'use_sqlite' => q[1],'use_sqlite' => q[0]," <MyConfig.pm >nosqlite.pm

Found bugs

I filed RT tickets for all found instances and keep the patches in my distroprefs which you can easily git clone into ~/.cpan.


Done on my desktop machine with 2.8Ghz, 4 cores, 16GB RAM and 15GB free space on my SSD drive. I used 3 cores in parallel, dividing the workload (~26835) by 3. The 4th core for my daytime work. The perl was with DEBUGGING, -g3 and no threads. An optimized perl would have been much faster to run, but would have not detected assertions and would disallow gdb stepping into harder bugs. The only hard bug I found was Data-Alias invalid realloc, so an optimized perl would have been better. At all I needed a week, but it should be doable in 2-3 days with an improved setup. I constantly ran out of disc space after night runs, and rebuilt some popular distros more than 10x. Installing them permanently and checking for discspace would save a lot of time. My distroprefs contains 128 yamls with patches, andy's CPAN prefs contain 449 yamls.

Restarts done like this:

$ wc -l allmods
26835 allmods
$ echo $((26835 / 3))

$ grep -n LastModule mods.1 => 4000
$ for m in `tail -n$((8945-4000)) mods.1`; do echo $m; jpan -t $m 2>&1 | tee -a log.cpan1 | grep "Possible precedence issue with control"; done

See Steffen's Steffen's CPAN explanation and tsee/cpanperlbranch_smoke for smoker setup. He had 25GB and 7 parallel processes on 4 cores.

p2 progress

I announced that I want to parse 50% of the perl5 syntax within this summer with p2. I think this goal is doable.

The last weeks since YAPC I spent most of my p2 time integrating libuv as external "aio" module. This implements asynchronous non-blocking IO and rudimentary cross-platform process support. libuv is essentially the node.js backend library, and MoarVM stated its goal to switch from apr to libuv also.

I needed libuv before the p5 syntax to check the ability to do fast and easy library bindings with lots of callbacks, and without a FFI yet. This gives me now the feeling how the FFI should be implemented, and if the current post-XS API is good enough or lacks some API support from core. As it turned out only some class (=type)/OO features needed to be added to the public potion API, the p2 backend. Everything else went smooth. I also fixed some more core bugs on my way.

I only added a rudimentary testcase, open a TCP connection with a connect callback, nothing realistic so far.

The libuv documentation is here: http://nikhilm.github.io/uvbook/index.html

The p2 aio binding is documented here in the doxygen docs and the htags API reference.

Now I continue where I left off at YAPC, where I added BEGIN block support: The expression parser, method and function calls, and global symbols.

About Reini Urban

user-pic Working at cPanel on B::C (the perl-compiler), p2, types, parrot, B::Generate, cygwin perl and more guts (LLVM, jit, optimizations).