Reviewing Perl 5 . in @INC at the Perl Toolchain Summit
Location: Lyon, France
Date: 12 May 2017
Attending: SawyerX, Merijn (Tux) Brand, Todd (toddr) Rinaldo, Nicolas (atoomic) Rochelemagne, Lee Johnson, Aaron (arc) Crane, Leon (leont) Timmermans, Matthew (alh) Horsfall, Kenichi (charsbar) Ishigaki, Graham (haarg) Knop, Karen (ether) Etheridge, Stefan (nine) Seifert, Aristotle
We met to discuss the recent changes to @INC for Perl 5.26.
A timeline and summary of CVE-2016-1238 was given to attendees. P5P has not yet disclosed the original report that led to removal of . from @INC in 5.26 but will be doing so soon. It was requested that this be delayed due to the severity of the original bug.
As of 5.26.0, Perl will be compiled without . in @INC by default. There is a Configure option (-Udefault_inc_excludes_dot) to revert Perl to its 5.24 behavior but this is not recommended.
We had a discussion about the state of CPAN with respect to this new feature. There was agreement that everything that could be done to mitigate the build/test/install failures had been done in preparation for 5.26. These were:
- Test::Harness now injects PERL_USE_UNSAFE_INC=1 into %ENV before executing unit tests.
- cpan clients (cpanm, cpanplus, etc.) now inject PERL_USE_UNSAFE_INC=1 when downloading and installing modules
- Mitigation has been done in perl to make it more clear why do "module.pl" fails because . was not in @INC. Special warnings will emit if do "module.pl" would have succeeded had . been in @INC.
There are 3 deprecations we identified needed to be discussed. The conversation quickly became confusing keeping them distinct.
- Drop support for the build option to compile a perl with . in @INC
- Drop support for the environment variable PERL_USE_UNSAFE_INC in the perl binary (adds . back into @INC when the program is executed)
- Drop support for injecting PERL_USE_UNSAFE_INC=1 into the environment during build/test/install
There was general agreement that the compile option (1) should be removed in 5.30. People who want it back can simply patch it back into perl with a couple of lines in perl.c
The next logical deprecation seemed to be to stop setting PERL_USE_UNSAFE_INC=1 in cpan clients and TAP::Harness. We felt it was a little early to determine when this might happen. However we did identify that we should add support for discouraging the build systems from setting PERL_USE_UNSAFE_INC=1 by a meta flag in META.yml: 'x_use_unsafe_inc'. Graham Knop (haarg) will be working on getting this functionality implemented on the CPAN clients. The cpanm patch can be found at https://github.com/miyagawa/cpanminus/pull/536.
Test::Harness will likely need to set PERL_USE_UNSAFE_INC indefinitely because it has to support older Perls.
Fixing CPAN
Module::Install has proven to be one of the hardest problems to work around. If you download a package and try to build/test/install it without first setting PERL_USE_UNSAFE_INC=1, then it will fail unless Module::Install is installed in your perl distro. The only solution to make this work better will be to re-release ALL Module::Install CPAN modules with BEGIN { push @INC, '.' } at the top of Makefile.PL.
There are 4700 distros on CPAN which use Module::Install. That's about 14% of CPAN which probably need to be re-released to fix this situation.
Adam Kennedy owns 220 modules. Many of which are well up river. He is inactive on CPAN but usually very open to sharing his modules for release fixes. Karen will be looking at taking over all of them to do a release to fix this.
Unit Tests the BSCINABTE way
The final point of discussion was the "correct" way to fix unit tests. A common pattern in unit tests was to assume that the test is being run from the base directory of the dist. Often people put the following in unit tests: use t::lib::MyTestModule;
Some work had been done by the group on this topic and for the most part, people were uncomfortable using the BEGIN { push @INC, "." } trick. Other options suggested were:
use lib "."; use FindBin; use lib $FindBin::Bin/t/lib; use MyTestModule;
An equivalent to blib was discussed as an idea. The code required to get blib to function correctly on all platforms is significant though. In the past one of the tricks was manipulating the __FILE__ string to add an entry to @INC.
No real conclusions were reached.
Action Items from the discussion
- Graham Knop (haarg) will be getting support for x_use_unsafe_inc implemented in the CPAN clients.
- Karen Etheridge (ether) will be working on patching ADAMK modules
- Todd will be starting a discussion with p5p about deprecating the build argument for Perl:
-Udefault_inc_excludes_dot
I blog about Perl.
You all know it was unaccountable bureaucrats in Europe making wild, life changing decisions without double checking with the impacted communities lead to Brexit...
"Adam Kennedy owns 220 modules. Many of which are well up river. He is inactive on CPAN but usually very open to sharing his modules for release fixes. Karen will be looking at taking over all of them to do a release to fix this."
I'll (STEVEB) gladly help out here if someone wants to give me co-maint on a bunch of them and let me know which ones.