Am I pandering to the smoke testers?
I have a quick sanity check I'd like to throw out there.
I've been working all week on Alien::TinyCC, and I made some great progress. My initial distribution wasn't properly checking that the build steps completed successfully, so I wasn't getting any useful information back from the smoke testers. Once I fixed that, I started to get some useful feedback. Mostly.
I have a Macbook, and I've been testing my build on my Macbook, so I had a reasonable expectation that my build setup would work on Macs. However, every single report from a Mac smoke tester was coming back "UNKNOWN", which basically means it failed at the build stage. How could this be?
It turns out that all of the Mac smoke tests came from Nigel Horne's Mac boxes. If you look at those reports, you'll notice the two reasons for failure are quite odd: error: unrecognized command line option "-fassociative-math" and error: bad value (native) for -march= switch.
These are odd errors because neither of these command line switches are mine. They come from the test box's environment variables. To be sure, I modified the Mac build process to check for and scrub those switches from CFLAGS and/or CPPFLAGS when appropriate (i.e. when using llvm to emulate gcc, and only on Macs). Doing this gets me past the build stage, and all the way through successful tests. That leaves me with two two lines of questions.
First, are Nigel's boxes able to build XS modules without trouble? I'm not sure because there's no way I could find to locate Nigel's other reports for other modules. But if so, does this mean that our toolchain actually sets the comipler environment variables? Or, perhaps, our toolchain doesn't even ask the system directly for its CLFAGS and CPPFLAGS? I suppose ExtUtils::MakeMaker may not even reference CFLAGS or CPPFLAGS, and Module::Build may not ask those environment variables for help, either. Should I have my Alien package ignore those variables in its Makefile, too? Or perhaps I should override them with Perl's settings for those flags?
Second, am I working to hard on this? On the one hand, I would say that modifying my build setup specifically so that it builds on this one smoke tester's possibly erroneous boxes seems like I'm pandering to the smoke testers to build my "PASS"es. But I'm not, really. No other Mac boxes were reporting anything. Unless I made these modifications, I wasn't getting any smoke feedback on Macs. I didn't even know if they passed their test suite because I couldn't get past the compile stage. Still, is this too much? Should I just email Nigel and ask if he can change his environment variables? Or, should I replace CFLAGS and CPPFLAGS with Perl's configure-based flags?
This is my blog about numerical computing with Perl.
Since the only Mac smoke tests you got experienced the problem, even if they were from a single tester, the steps you take scrubbing those values seem reasonable.
I'm not an XS developer though. You might try reaching out to the cpan-testers-discuss list at http://lists.perl.org/list/cpan-testers-discuss.html
> am I working to hard on this?
No. If you contact tester and investigate why those switches are in place, and then try either fix your distribution or fix tester's box, that would be hard.
Nigel, I am told, runs his tests on a softly-mounted NFS file system, and this has many differences from a POSIX system, some not so subtle. It bans the creation of executables for one, and this makes C compilation pretty much impossible.
Why this creates your reported symptom I do not know, but I do not believe any Perl distribution which requires compilation of C does or can work in Nigel's setup. It's unfortunate that there seems to be no easy way to detect when a non-POSIX FS is in use, so that you can simply bail.
@Jeffrey -
Actually, I *can* run the compiled tcc executable. The test suite invokes a great many C test programs through tcc, which is a binary built by the distribution itself, so it looks like Nigel's setup does indeed allow executables to be built.
@Ben -
I even tried searching cpantesters using google with the search "Nigel Horne site:http://www.cpantesters.org", and that did not come up with any test reports, only reports about Nigel being one of the top testers for many various months. I suspect this is because the robots.txt file forbids the indexing of those pages. If we could simply convince them to allow Google to index the test reports, that would be enough to get started. An additional functionality to search for various testers for various sorts of modules would be even better, but quite a bit more work, I suspect.
Sorry, I should have added more links to that discussion. I brought up the issue of CPAN testers not being searchable using Google on the mailing list for CPAN testers a while ago using an example of a problem with obscure XS error reports. Barbie responded to the mail I'd sent on blogs.perl.org but it was just about the problem I'd had rather than the general problem of not being able to search the test reports. If other users chimed in with other examples of how they would like to be able to search through CPAN testers' reports, then it probably would make a better case.
@David: Interesting. For Marpa::R2, autoconf reports as follows:
And that is the failure message for me on Nigel's reports (http://www.cpantesters.org/cpan/report/dabce4b2-104a-11e3-83fe-bae8681aeb38). Ironically, my primary development platform *is* Darwin. Since all other reports from Darwin are success's, I was happy to believe the failure was a peculiarity of Nigel's platform.
Starting a lynch mob is unlikely to win you any favours.
Please read the reply to your post carefully. It does answer your question. In particular the third paragraph.
By all means go ahead and write a CPAN Testers search engine. But I get the feeling from all these posts recently, everyone wants someone else (usually me) to do the work.
Thank you for your hard work in setting up the CPAN tester's site, which I very much appreciate.
My point was actually that my original query about not being able to search for the obscure XS errors quite possibly did not make a very convincing case for allowing search of the CPAN tester's site, and that perhaps a better example of the utility of search was needed.
I don't know how to do that, I was informed that the database available on the website does not include the reports themselves. How can I access the database containing the reports?
"If you build it they will come".
You already have access to the text based reports, and the structured data is also described (see CPAN::Testers::Report, CPAN::Testers::Fact::LegacyReport and CPAN::Testers::Fact::TestSummary). Using test examples you can create a demo search site.
If you plan to host this yourself, David Golden can provide you with keys to access the metabase, from which you can obtain the structured reports. However, as the db is now so large, it will be better to run this on one of the CPAN Testers servers. Probably the metabase server. There is ongoing work to move the current setup to a new server, using MongoDB for storage, so that would be an ideal place to have the kind of search engine you want.