The topic of the week on P5P was the clean up and release of perl 5.18.0 which was released on 5-18 for North Americans. RJBS in a seperate email said blead would be reopen for patches on Tuesday, as 5.19.0 is scheduled to be released on Monday (May 20).
Congratulations to the perl5 team for this new release!
]]>Topics this week include:
Fire up perlbrew: 5.18.0-RC2 is here. Time to test all your modules
against this release candidate! The perldelta also makes great
reading - all the stuff that's new and different. Some highlights
include the hash ordering changes, experimental lexical subroutines,
Unicode 6.2 support, given aliasing global $_ and a metric
ton of bug fixes in the /(?{})/ and /(??{})/ regex constructions.
Proposal: initial type annotation in SV
Sébastien Aperghis-Tramoni wants to record the initial type of an SV to help perl data serializers (Storable, Sereal, JSON::XS, etc) properly encode Perl data structures.
What does PERL_HASH_SEED do?
Just in time for perl-5.18 Yves Orton committed patches to make the environment variable PERL_HASH_SEED (and some new helpers) change the behaviors of hash storage and traversal.
Designing an API
Responding to Nicholas Clark's email pointing out that XS doesn't have a 'real' API, Yves Orton asks, should there be one?
Should we make hashes return keys in FIFO order?
Not quite done with hashing, Yves Orton asks if perl in 5.20 should return keys in FIFO order (i.e., return keys in the order they were inserted.) Some love it, some hate it; standard P5P reaction. :)
Release timing drift
David Golden points out that perl releases seem to be drifting further back in the year than earlier yearly releases. He proposes moving freezes forward a month so blockers can be resolved with less urgency.
]]>Topics this week include:
Karl Williamson reports on a problem in Perl's regex engine which he's made a patch for. This thread goes over the specifics and asks RJBS how the patch ought to be applied to blead for 5.18.
How on earth did we manage to break pack() so badly
Yves Orton writes, um, a strong critique of the behavior of pack() since 5.10. He feels it's annoying and wrong. Dave Mitchell thinks it might need some tweaks but is basically proper behavior.
DAVEM TPF Grant April 2013 Report
Dave Mitchell gives his monthly summary of work done on perl as part of his TPF grant. These are always interesting reads if you like to track the history of perl bug detection and squashing.
Problems with Carp::longmess() in 5.17.10 and before
Watch live code archealogy as P5P digs deep into some messy internals of Carp.
What does PERL_HASH_SEED do?
Steve Hay asks about the variance between the documented behavior of this environment variable and the observed behavior in blead.
]]>Topics from this month include:
Welcome to Perl 5 Porters Monthly, a summary of the email traffic of the perl5-porters email list.
Topics from this month include:
Welcome to Perl 5 Porters Monthly, a summary of the email traffic of the perl5-porters email list.
Topics from this month include:
]]>Welcome to Perl 5 Porters Monthly, a summary of the email traffic of the perl5-porters email list.
Well, that was a nice little break. I didn't intend for it to be so long but there you go. Life gets real sometimes. To catch up I'm basically going month by month through most of April. I plan to resume monthly summaries starting the week of April 29, 2013.
Topics for this month:
These are all of the "read the thread" link variety.
]]>Topics for this week include:
Alex Vandiver posted a message that cPanel has shipped a version of Storable identified as 2.39_01 which appears to break the vendor-installed perl environment on cPanel installations.
This has the effect of making other co-located Perl applications (RT for example) break. A back-and-forth about this issue revealed that cPanel is preparing a fix for this problem to be released in January 2013 - it looks like they are moving to Sereal or Data::Messagepack.
Separate fixes to spelling from changes to code layout in patches
Earlier in this week, a patch was committed with included some whitespace changes. In a followup, Ricardo Signes, the perl project manager asked people submitting patches to
please separate fixes to spelling and grammar errors from changes to
code layout.
Term::ANSIColor 4.01 released
Russ Allbery announced that there's a new release of Term::ANSIColor which has been in Perl core since 5.6. This release includes improved docs and fixes some test cases.
]]>Topics this week include:
Dave Rolsky announced that perl-5.17.7 is now available from a CPAN mirror near you.
Solaris smoke testers wanted
Dave Rolsky also encouraged people who use Perl on Solaris to join the CPAN testing/smoking group.
Find out how to set up a smoker box
Parrot 4.11.0 "All together - Happy Birthday Lovebird" Released!
Reini Urban announced that Parrot 4.11.0 is now available for download.
successful s///e clobbers $! on MSWin32
Which makes bugs on Win32 harder to track down when using Carp because Carp::format_args() does such a substitution.
JROBINSON grant report, #13
Checking in on Jess Robinson's Perl to Android cross compiling effort, it looks like there's an interesting side project which allows a Java process to schlep fatpacked perl modules to an Android platform.
Full details are in the grant report. A demo is available on a blog post.
POSIX wrapper for strptime()
LeoNerd proposed making POSIX/strptime unbroken cross-platform, but there was quite a bit of pushback on the idea. The primary objection was that the purpose of the POSIX module is to expose OS-specific functionality, and the second objection was that a cross-platform strptime already exists in Time::Piece.
Is this a known unicode bug in 5.16.2?
Anders Melchiorsen reported a strange bug in Perl's regex engine when it somehow thinks it's processing UTF-8 (even though it isn't.) A series of messages tracing the bug and trying to understand its origin followed. In a followup Dave Mitchell supplied a fix for the problem.
]]>Topics this week include:
Peter Martini proposed adding a 'prototype' attribute into the Perl core, so that other modules can use the syntactical space more cleanly. If you're interested in the specific implementation notes, please read the thread.
Undocumented %ENV behaviour
Kent Fredric reported a surprising property of %ENV in blead. Apparently, a value stored in an environment variable is stringified. Kent later submitted some patches to create a warning class when a ref is inserted into %ENV (that would be stringified.)
rand() on Windows only uses 15 bits of entropy
You may recall a thread from a few weeks ago pointed out that the rand() call on Win32 was woefully not random. In response, Yves Orton recently created some patches to add the TinyMT32 random number generator. All of this work is still very experimental, but there is going to be a round of "does this break CPAN?", so it will be interesting to see the outcome of those tests.
]]>Topics this week include:
Ricardo Signes published a security notice about the Storable module that ships in Perl's core. He writes:
A number of times over the years, there's been discussion about Storable
as a vector for attack. If a user can feed you Storable data that you
didn't expect, he has a good chance of doing nasty things to your
program. This has been discussed on p5p and at YAPCs, but sadly never
made it into the documentation.
This has been fixed[.]
Thanks to Brian Carlson of cPanel who brought it to the P5P security team's attention.
security notice: Locale::Maketext
Rik also issued another notice courtesy of Brian Carlson about the
Locale::Maketext module which also ships in Perl's core. Ricardo
summarizes the problems as:
* in a [method,x,y,z] template, the method could be a fully-qualified
name
* template expansion did not properly quote metacharacters, allowing
code injection through a malicious template
Does Unicode mandate a collation order?
There was a question (not strictly Perl related) about whether Unicode
mandates a collation order. It does, and the precise answer, according to
Tom Christensen is determined by what DUCET value is used internally by
the Unicode routines.
If you want the full explanation, read Tom's reply
a job for someone? SvREFCNT_dec_NN()
Dave Mitchell writes that he's just added the SvREFCOUNT_dec_NN() macro
which can be used in situations where the SV cannot be null. In such cases,
the macro saves some code space, a test and a jump in the code path. There
are 500 uses of the original SvREFCOUNT_dec in perl.
If you ever wanted to dive into Perl's core and were looking for a fairly easy task to get you going, this might be what you're looking for.
CERT Perl Secure Coding Standard
Yves Orton wonders if anyone's seen CERT's Perl Secure Coding Standard. Jeffrey Thalhammer writes back that he's hopeful a Perl::Critic module will ship early next year incorporating the standard's guidelines.
]]>Topics this week include:
If you ran this program on your system what would you expect the results to be?
my %nums;
my $cnt = 0;
foreach my $i (1 .. 1_000_000) {
my $num = rand;
$cnt++ if ($nums{$num});
$nums{$num} = 1;
}
print "$cnt out of 1,000,000\n";
If you run it on Windows, you might be surprised to discover that it will
print 967232 out of 1,000,000 every time. That means there's only 32,768
possible floats between 0 .. 1 that rand() will generate on that platform.
Other platforms don't seem to be affected by the lack of entropy in rand().
This comes from a RT ticket opened by Brendan Byrd. There was a bunch of discussion about this. Ricardo Signes indicated that he's very interested in seeing an implementation of the Mersenne Twister algorithm implemented, which follows some suggestions by Nicholas Clark.
RFC: Removing several undocumented functions from the Perl core
Karl Williamson suggested that several undocumented functions in Perl's API be removed for 5.18. They are:
Nicholas Clark suggested the calls be flagged as deprecated in 5.18 and removed in 5.20.
5.18 VT in \s
Karl Williamson suggested that the vertical tab character, added experimentally to m/\s/ in 5.17 be formally released in 5.18. He notes that this addition caused no perceptable failures in CPAN smoke testing and doesn't expect any failures from releasing it into the wild, partially because the vertical tab character is so rarely used.
For the full context about vertical tab's inclusion, read the thread.
Comment period extended for Unicode's changing some common characters from Punctuation to Symbol
Completing his trifeca on this week's summary, Karl Williamson noted that Unicode has extended its comment period to January 21, 2013 for changing some common characters from the "punctuation" class to the "symbol" class.
You can make comments by visiting here.
David Golden asked in a followup the following questions:
* How can we better document (if we're not) the forward compatibility
risks inherent in using Unicode character classes?
* How can we let programs introspect the version of Unicode that Perl
provides?
* Is it possible to make any of this pluggable, so a program could
specify which version of Unicode classes they want to use?
Karl replied that it's not pluggable but it is possible to compile any perl
that supports Unicode with a version specific character database by
following the instructions in README.perl inside of lib/unicore if you
wanted to downgrade a particular perl for some reason.
He also notes that these proposed changes were included last summer
experimentally and only broke 1 CPAN module. He says that the [[:punct:]]
class matches both the Unicode ASCII-range symbols plus punctuation, so most
Perl programs will not notice any of these changes.
With respect to the third question above about Unicode version inspection (programmatically), Karl wrote:
This information has long been available through Unicode::UCD::UnicodeVersion()
This week's topics include:
Ricardo Signes announced that perl-5.17.6 is now available on a CPAN mirror near you.
Read perldelta and see what's new
On deprecating unescaped literal left brace
Karl Williamson summarized a discussion from P5P several months ago about literal left braces (i.e., "{") in regex. He writes:
The motivation for doing the deprecation is to eventually forbid an
unescaped literal left brace, and the motivation for doing that is
really threefold:
a) It is too easy to make a typo in a quantifier, and have it silently
be turned into a literal. An example is the programmer thinking that /x
allows spaces in the quantifier, like "/foo{1, 3}/x". This matches the
characters literally. There are similar examples, like forgetting the
closing brace.
b) We are prevented from extending the quantifier syntax to, say, allow
such a space under /x, or to accept "foo{,7}", which now also silently
matches literally.
c) The left brace (or curly bracket) is the obvious candidate to use to
extend the language in various ways. For example, in Unicode there
several different types of word boundaries that could reasonably be
desired. Perl only knows about its traditional one. It would be nice
to be able to specify a different one, like "\b{g}"
The reason there is a problem with blead, is if the delimiters of the
pattern are braces, the lexer/tokenizer strips off any backslash that
the programmer added before handing it to the regex parser, so it is
futile for the programmer to add a backslash. There is no work-around,
except to not use braces as the delimiters. Braces are used very
commonly as delimiters.
[...]
That presents a way out of the dilemma. We revert the patch that
instituted the current warning. Then we deprecate adding extraneous
backslashes, so that "m{^a\{1,2\}$}" would generate a warning. In 5.22,
we would change the lexer/tokenizer to not strip off the preceding
backslashes of paired regex delimiters, and reinstitute the warning
about unescaped left braces. We announced in 5.16's perldelta that
unescaped left braces would change behavior in 5.20. We should update
that announcement in the deltas for 5.18 through 5.22 that the release
for this change is 5.24.
Yves Orton and Ricardo Signes seem to think this is a good change.
Parrot 4.10.0 "Red-eared Parakeet" Released
Reini Urban announced that Parrot 4.10.0 has been released.
Perl in Git on Windows?
Leon Timmermans responded to a post on blogs.perl.org about the bundled perl in the "Git on Windows" installation bundle. Apparently it's a rather old version of Perl (5.8.8). Leon thought it would be nice if the Perl core could support msys out of the box and asked if anyone was interested in porting the build patches into this decade.
Move tests not using t/test.pl into a separate directory
James Keenan describes an unfortunately common scenario when someone refactors a bit of testing code in perl's core, and is subsequently told after submitting the patch that it's an invalid change for some reason.
His proposal is to document and sort the test code in core so that a contributor can understand what testing requirements and assumptions may be used to refactor it in the future.
He wants to:
* Create a new subdirectory called t\/opbasic.
* Move any t/op/*.t files which do not already use t/test.pl into
t/opbasic -- i.e., assume that any t/op/t file which already uses
t/test.pl does so properly.
* Adjust 'perlhack' and other documentation as needed.
* Study those files which we have moved to t/opbasic to see whether they
are, in fact, eligible to use t/test.pl. If so, move them back to t/op
and convert them.
And as a corollary, encourage new contributors to add descriptions to
the files under t/op, as we did in St Louis this past weekend.
Hash randomisation breaks CPAN
Father Chrysostomos posted a list of CPAN modules which break due to hash order assumptions which became truly random recently in blead.
The modules are:
(And there are more discovered too. See the full RT ticket for all of them)
Nicholas Clark writes:
This seems an excellent task that *anyone on this list* could help chip away
at, given that likely all it needs is a knowledge of perl.
So, volunteers welcome.
1) Pick a module from the list
2) figure out the fix
3) send a patch to that module's bugtracker
4) feel good
This week's topics are:
Dominic Hargreaves announced that perl 5.12.5 is now available on a CPAN mirror near you. He writes:
This release is primarily a security fix release, according to the
maintenance policy for Perl 5. Three security fixes are included:
- Encode decode_xs n-byte heap-overflow (CVE-2011-2939)
- File::Glob::bsd_glob() memory error with GLOB_ALTDIRFUNC (CVE-2011-2728)
- Heap buffer overrun in 'x' string repeat operator (CVE-2012-5195)
More details about the release can be found in the perldelta document.
no easy link to quick hacking recipe?
Dave Mitchell proposed adding a section in perl's README about submitting quick patches to Perl's source code, since there doesn't seem to be a direct link to it, although it turns out there's a section entitled "SUPER QUICK PATCH GUIDE" in perlhack.
-DNO_TAINT_SUPPORT in blead
Steffen Mueller announced he'd pushed his no-taint patch to blead, although there is currently no direct support for it in the Configure script.
]]>This week's topics are:
Ricardo Signes announced that perl 5.16.2 is now available on a CPAN mirror near you.
perl-5.12.5 RC1 is now available
Dominic Hargreaves announced that perl 5.12.5-RC1 is now available on a CPAN mirror near you.
What happened to the whole "small core" idea
Last time, Peter Rabbitson asked what had happened to the idea of making Perl's core smaller. Ricardo Signes, the project's arbiter of taste, finally replied this week. I encourage you to "read the whole thing" as all the cool kids say these days, but here are some good excerpts:
Later on in this thread, you say something like, "people are talking about
getting [subroutine signatures] into 5.18." I never thought this was
seriously considered. That is, I thought maybe some were hoping to see
it become an experimental feature, but not anything we'd be stuck with
once we proved it stank.
...but really, I didn't think it was even going to get that far. But did
it have to become a discussion about that? I didn't think so. There were two
important things to figure out: (a) how can we get perl equipped with hooks
for signature systems that share a common underpining so we can figure out what
a good "core" one might be and (b) what might that core one be?
(a) would be nice to see, at least experimentally in 5.18
(b) seems like it will be an ongoing discussion; why not have some of it on
p5p? and if there's an implementation that is built in a branch, without the
benefit of the hooks suggested by 'a' that lets us play with the things as
discussed... awesome!
[...]
As you noted, the bikeshedding is largely about design decisions that have
*not* been proved on CPAN or become a standard. This is a warning sign that
it's maybe not baked enough. I don't think that means that we should stop
talking about it, or stop building on the code that implements it. That
currently-in-a-branch-of-perl code seems pretty good, and may very well be the
code that does evolve into something proven enough to become the beloved
reference implementation of signatures. After all, it's from that code that
the signature-adding APIs will probably be born, right?
Eliminating the "rehash" mechanism for 5.18
Yves Orton proposed eliminating perl's defensive mechanism against pathological hash insertion attacks and replacing it with a per-run random hash seed. He explains:
Choose a random hash seed at process start up as it effectively prevents
the attacker from constructing their colliding keys remotely. However
it also means that the order in which a set of keys are stored in a hash
becomes random.
[...]
Besides the advantages of avoiding the costs of the rehash mechanism
this change will smoke out any hash order dependency bugs in the
modules on CPAN, and in our own code. (I found at least two unrelated
bugs in code in core due to hash randomization.) It will also prepare
the ground for people to choose alternative hash algorithms for Perl
to use, or to make it possible to use specialized hash algorithms in
certain contexts such as 64 bit builds.
The downside however will be that it almost certainly will make lots
of lazily written code to fail. But IMO the cost is worth it.
Most people thought this was a good idea, but some thought there were additional security precautions which should be considered.
benchmarking Murmurhash3 vs One-At-A-Time as perl's hash function
Yves Orton had another interesting hash related email this week. He'd written earlier in the summer that he was looking at replacing perl's hash algorithm with one that could offer some signficant performance benefits especially with longer keys. This week he reported the results:
Conclusion: on my laptop Murmurhash3 is more than 3 times faster at
hashing longer strings than one-at-a-time, and roughly the same for
shorter strings.