Smoking CPAN in one line

By Reini Urban on

I'm using CPAN::SQLite. I don't like it and I'm trying to improve the locking issues with concurrent process all trying to read from cpandb.sql by writing a new CPAN::UnQLite. But this is something else.

Today I want to smoke whole CPAN in one line to check the new experimental return-or check by Niels Thykier for "Possible precedence issue with control flow", for which we filed several bug reports in the last two days. See https://paste.debian.net/16932/ and perl #59802. To repro add this patch from Niels to latest blead, if it's not already included.

$ cd ~/.cpan
$ alias p='perl5.19.2d-nt@1292fdb2'; alias ppan='AUTOMATED_TESTING=1 PERL_MM_USE_DEFAULT=1 PERL_EXTUTILS_AUTOINSTALL=--defaultdeps p -S cpan'
$ cat allmods.sh
#!/bin/sh
sqlite3 cpandb.sql 'select mod_name from mods group by dist_id order by mod_name;'

$ for m in `./allmods.sh`; do echo $m; ppan -t $m 2>&1 | tee -a log.cpan | grep "Possible precedence issue with control"; done

And in a 2nd term watch it flow, and react on keyboard input. 

$ tail -f log.cpan

Note this is just a oneliner, not a real smoker, which needs to setup some environment variables and disable STDIN. For smokers see http://wiki.cpantesters.org/wiki/QuickStart

Addendum, keep some dependencies

cpan -t does only testing, but not installing. Major dependencies get built and tested all over again, if they are not installed. This costs some unnecessary minutes. So I added this in a seperate term to keep the most prominent deps.

$ alias jpan='ppan -j ~/.cpan/CPAN/nosqlite.pm'
$ jpan `grep -a "Warning: prerequisite" log.cpan|cut -d' ' -f3|sort |uniq -c|sort -rn|perl -alne'print $F[1] if $F[0] > 5'`'

I need to use a version without use_sqlite to avoid the CPAN::SQLite locking bug. Yes, I've tried to fix it, but some writing codepath still holds a reference to a stmt handle which prevents the readers to be shared.

cd ~/.cpan/CPAN/
sed -e"s,'use_sqlite' => q[1],'use_sqlite' => q[0]," <MyConfig.pm >nosqlite.pm

Found bugs

I filed RT tickets for all found instances and keep the patches in my distroprefs which you can easily git clone into ~/.cpan.

Setup

Done on my desktop machine with 2.8Ghz, 4 cores, 16GB RAM and 15GB free space on my SSD drive. I used 3 cores in parallel, dividing the workload (~26835) by 3. The 4th core for my daytime work. The perl was with DEBUGGING, -g3 and no threads. An optimized perl would have been much faster to run, but would have not detected assertions and would disallow gdb stepping into harder bugs. The only hard bug I found was Data-Alias invalid realloc, so an optimized perl would have been better. At all I needed a week, but it should be doable in 2-3 days with an improved setup. I constantly ran out of disc space after night runs, and rebuilt some popular distros more than 10x. Installing them permanently and checking for discspace would save a lot of time. My distroprefs contains 128 yamls with patches, andy's CPAN prefs contain 449 yamls.

Restarts done like this:

$ wc -l allmods
26835 allmods
$ echo $((26835 / 3))

$ grep -n LastModule mods.1 => 4000
$ for m in `tail -n$((8945-4000)) mods.1`; do echo $m; jpan -t $m 2>&1 | tee -a log.cpan1 | grep "Possible precedence issue with control"; done

See Steffen's Steffen's CPAN explanation and tsee/cpanperlbranch_smoke for smoker setup. He had 25GB and 7 parallel processes on 4 cores.

1 Comment

Author Profile Page James | July 20, 2013 10:33 PM

SQLite shouldn't be using exclusive locking for multiple concurrent reads, only writes, surely? Wouldn't excessive locks in SQLite mean time to a database system with a server, not another serverless solution?

About Reini Urban

user-pic Working at cPanel on cperl, B::C (the perl-compiler), parrot, B::Generate, cygwin perl and more guts, keeping the system alive.

More info »