Overview of current maintainer fails

schmorp recently complained about a maintainer fail in Math::BigFloat, and since this is not an isolated incident of the last years, I try to give a summary of all the maintainer fails of important modules lately, as I happen to fix them privately.

Problems in core ext/ and dist/ modules, maintained by p5p:

These are all doomed modules, as p5p refuses to take bug and security reports, and most of them are not maintained and fixed and improved in cperl only.

Math::BigFloat

Maintained by p5p, responsible: pjacklam

regression in blsflt (left shift) and refusal to fix it, lack of understanding of float math.

fixed in cperl only. in cperl we also heavily modernized all the bigint/bignum/bigrat/Big::* modules to use typed signatures.

there were several podcheck fails, which were promptly fixed. so they do react fast, their only problem is the lack of understanding.

Storable

maintained by p5p, responsible: nobody

missing features, grave security problems with bless and tie, other security problems on 64bit with overlarge hashes and names. fixed in cperl only.

tonyc recently replied to a similar but limited bugreport with a partial fix, which doesn't really help.

Net::Ping

unmaintained by p5p, responsible: nobody, previously smpeters

p5p took it over from him, while smpeters refactored the test suite for a while. many bug reports and patches still sitting in the cpan queue, no idea about the p5p queue, because they blocked me there.

development stalled, so I added most needed features and fixes. fixed in cperl, and on https://github.com/rurban/Net-Ping

ExtUtils::Constant

unmaintained by nwclark, responsible: nobody

Development and maintenance stalled, many fixes unanswered. I had to publish all fixes to cpan as UNAUTHORIZED RELEASE.

Fixed in cperl.

Opcode

unmaintained by rgs

Development and maintenance stalled, many improvements ignored. had to publish my own version Opcodes. Maintainer only wants Opcode as parent module for Safe, but not to specify Opcode properties. This is hardly an acceptable point of view.

POSIX

maintained by p5p

Basically this module is only plagued by old sins, exporting all symbols at once and not by request. But the maintainers obviously don't know about this and accepted a bugreport that a new symbol is not automatically exported. This is a feature, not a bug.

The unsafe tmpnam is still exported, but at least deprecated. The new tmpnam replacements tmpfile, mkstemp, mkdtemp are missing.

POSIX::1003 should be used instead.

Problems in core cpan/ modules, maintained by various:

autodie

maintained by pjf, several private core fixes, several unfixed problems

this is a stable pragma with a complicated scoped feature set, which does not really work in corner cases. it broke the compiler with gensyms and empty stashes, and it broke Porting/sync-with-cpan. no autodie does not properly restore system() from CORE::. I had to remove autodie from cperl's improved sync-with-cpan, and I urge everybody else not to use it, as the errors are often invisible and undetected.

It is also considered bad practice to use it. You need to know which open can fail and which not. It's recommended for complete newbies, such as use warnings and Carp, which introduce heavy memory regressions.

Socket

maintained by pevans, but this guy can usually be trusted.

The queue is quite small.

But I had to add missing IPv6 constants in cperl for Net::Ping, a lot more are still missing for ping6 functionality. http://search.cpan.org/~rurban/Socket-2.021_01/

JSON::PP

unmaintained by makamaka

unwilling to support Cpanel::JSON::XS the follower to the broken JSON::XS

unwilling to support cross-compatibility with the other JSON modules, throws several redefinition warnings if the loading order is different.

unresponsive. we typically wait >1 year to get answers on feature requests. last release 2013.

in cperl we fixed overload stringify and eq as in Cpanel:JSON::XS esp. "true" => true not 1

Pod::Perldoc

now maintained by mallen

not much problems, but horrible perldoc.3 vs perldoc.1 conflicts in core integration.

podlators

horrible core integration of Pod::Man and a broken $Maintainers{MAP}

Pod::Simple

Properly maintained, but they still think they are alone on this world, same problem as with Ingy. That there is only one installed perl, and only this can run the tests. There exists $^X for a reason.

I fixed their t/search50.t test in Dec 2015, but it is still not handled.

Since cperl will ship its own heavily modernized version, it's no big deal. signature all the methods and get type violations at compile time, which caught many errors already.

ExtUtils-MakeMaker

maintained by the toolchain gang

I don't expect the toolchain folks to support the cperl variant, so I do it by myself, and added various new features along the line.

wrong Config dependencies, support PERL_USE_UNSAFE_INC=1, call darwin dsymutil with DEBUGGING, use the fixed and faster YAML and JSON cperl core parsers. see below at Problems in core toolchain cpan modules, kept fixes private.

Test-Simple

maintained by exodist, very sceptical

The big Test2 feature he is working on for 2 years is event-based streamable tests, which still is 2x slower so far. So a no go. See http://search.cpan.org/~exodist/Test-Simple-1.302014_007/lib/Test2/Transition.pod

The old implementation is too heavy, we simplified and improved it internally. Since cperl will ship its own heavily modernized version, it's no big deal. We improved many abstraction methods. Signature all the methods and get type violations at compile time, which caught many errors already. Some of our type checks will break your tests, esp. that the skip count needs to be a number.

Digest::MD5

maintained by gisle, no response

core had traditionally a wrong U32_ALIGNMENT_REQUIRED probe in Configure.

I added a proper probe to fix the intel architecture, and added compatible SIGBUS on alignment fails, to detect previously undetected alignment errors. Fixed 64bit and darwin multiarch probes. Fixed RT #77919

https://github.com/rurban/digest-md5/tree/intel-align-rt77919

Devel-PPPort

unmaintained by alh

Had to fix 5.16 binary names by myself, which should have happened in the last 8 years, as p5p pushed for binary names, and I opposed it.

cperl also supports @INC without . (-Dfortify_inc)

See https://github.com/rurban/Devel-PPPort/tree/516gvhv

Scalar-List-Utils

maintained by pevans

Many core features are improperly supported, and many features are missing, so people had to come up with List::MoreUtils. The current maintainers do not known about MULTICALL and lexical $_, the most trivial features needed for this module.

The recent div by zero fix was done much worse than in cperl.

At least he is responsive.

Config::Perl::V

Only minor issues. This is the only core module published as .tgz, so it cannot be updated by p5p with sync-with-cpan. only fixed in cperl.

Problems in core toolchain cpan modules, kept fixes private

Basically all toolchain modules which lack cperl support. Esp. the 'c' suffix version detection and comparison, and the lack of knowledge about cperl builtin modules without .pm: XSLoader, DynaLoader, strict, coretypes. The old toolchain still wants to install a missing XSLoader requirement, because it searches for modules statically, not dynamically.

CPAN-Meta-YAML, CPAN-Meta-Requirements, version, CPAN, CPAN-Meta, Parse-CPAN-Meta, EU-MM

The biggest problem is over-architecture, as the toolchain is typically the gathering place for all the important people too incompetent to do core. The old master Dan Golden is the only light in the dark.

Another problem is the lack of efficient JSON and esp. YAML support, and lack of YAML compliance. They gave up on YAML instead of fixing it, as we did.

With cperl we added fast XS variants for JSON and YAML, fixed the broken and overly strict YAML::XS, and use this throughout. And not the infant YAML::Tiny, which also deviates from the YAML specs. cperl added JSON and YAML validation tests for all existing JSON and YAML modules.

Problems in other cpan modules, maintained by various:

I've looked over all my >100 distroprefs patches, and in the end problems are only with ether and schmorp. schmorp at least knows what he is doing and eventually comes up with fixes by himself.

Moose

maintained by ether

Moose was compilable before ether took over, since then it eroded. The maintainer doesn't have an idea how compile-time vs run-time works, which should be critical understanding for such a compile-time optimized module.

A lot of compile-time code is not run in BEGIN or CHECK blocks, rather as main_cv code of compile-time imported modules, which does not work when separated into compiled and run-time.

I didn't even bother to file a ticket, as 1st compared to Mouse Moose is totally unusable, and 2nd the maintainer might be willing to fix it, but is not able to maintain this.

Sub::Name

maintained by ether

The new maintainer is not able to come up with 5.16 support, binary names. She refused to merge my fixes for several years.

B::Utils

maintained and taken over by ether. I suspect a hostile takeover.

The new maintainer was not able to maintain API stability for OP::parent, one of the biggest reasons this module exists, and lied for the reasons to take it over. A complete disaster. I had to publish my own stable B::Utils1 module.

Summary

So in summary, only p5p and ether are failing to do their job.

With the core cpan modules there are some typical outliers, nothing dramatic. cpan itself looks very healthy, whilst p5p looks totally dead.

Storable security problems and overlarge data

Some people might be aware that we (not p5p) fixed Storable already for the most egregious security problems with Storable, with public CVE, explanation and metasploit module, but those bugs are still in Storable as on CPAN and in the latest perl5.24.

Now a new problem arose on the horizon, that p5p dares to fix another bug, the inability to store overlarge strings. Which would be okay if they would have taken our fixes, which fixes the security problem and all overlarge data >I32, which are arrays and hashes also. But they didn't, so I have to complain again. People apparently don't like me to complain about p5p mistakes, but you should be concerned what p5p is doing.

See perl #127743 for the wrong fix, adding 2 new ops, where only one is needed, and this fix only fixes a third of the overlarge problem, and not the security problems. There's no need for them to deviate from the API with a worse fix.

And compare that to the real fixes at cperl, which I sent to Tony some time ago, but apparently with no effect. So they are playing dumb again.

Note that this is now a security problem they are avoiding to fix.

cperl #120

cperl #121

Note that I cannot fix that for you on CPAN, as Storable is maintained by p5p. It's only fixed for us. A fair warning.

And for the ones who are wondering: I'm not able to even post the fixes to the proper ticket and list, as they blocked me there. The reason for the block was entirely fabricated, normal censorship to silence criticism.

The removal of the lexical topic feature in 5.24

Now that p5p removed the broken lexical topic feature in 5.24 it's time to look back how that happened and why it should not have happened, and how cperl fixed that problem.

In 2013 Rafael Garcia-Suarez posted to p5p a plea to Salvaging lexical $_ from deprecation arguing that the recent deprecation of my $_ is wrong, and can easily be fixed.

He explained the problems pretty nicely on the language level, but forgot to explain the internal advantages of a lexical $_ over a global or localized $_. To add that it's pretty simple: A lexical $_ is an indexed slot on the pad array, calculated at compile-time, whilst a local or global $_ is a hash entry in the namespace. It's much faster to lookup, write and restore. The slot index is compiled and carried around in the op. That's why it's called lexical, it is resolved at compile-time, and not at run-time.

The best technical reason why a lexical $_ is needed is given by davem, the only smart guy in the room here at RT #94682. given uses an implicit my $_ internally, and every match uses it.

If you try to read the answers by p5p you will get headaches as their explanations are not only clueless and wrong, they even argue that using $_ lexically is semantically wrong! Eg. rjbs: "I think it is half-baked, confused, and confusing. I don't see how it can be made useful or straightforward, at the language level."p5p

He clearly is confused because of the tricky bugs on the ops he is talking about, but here he was arguing that the semantics are broken(!). Using $_ explicitly or implicitly; or global, local or lexical makes no difference, only to RJBS and to some other porters. Doy for example argues that my $_ should be removed because this example is broken:

my $_ = 'foo'; print = any { $_ eq 'a' } qw(a b c)

which I fixed in 10 minutes by using the lexical $_ in the XS implementation and not the global. See the simple patch at RT #113939

The underlying problems with the implementation of my $_ were quite trivial to fix. I just had to fix the initial check for OPp_TARGLEX in the cperl commit 597e929c9 August 2015, which I published quite well. But the message obviously didn't arrive at the p5p team, because the decided that my $_ cannot be fixed and needs to be removed end of August. This time without any deprecation period.

One other interesting new answer was by zefram that this bug if not fixable, because we'd need support for lexical $_ in every op which supports implicit $_ or takes blocks, such as grep and map, obviously not knowing the codebase that this is exactly what we were doing quite efficiently since 2001. There is e.g. a special lexical grep bit. A git grep GREP_LEX would have helped, which was added 2004 as a special bit to grep.

I think there's no way forward that looks at all similar to what we have. Any lexical topic variable would at least have to be selected on a per-construct basis: you can't have "my $_" causing grep and the like to use a lexical topic. A spearate "grep_lexical" operator should be fine, but if we go that route we should consider forcing the lexical topic variable to have a name other than "$_" to avoid the confusing shadowing. Actual "my $_" should be no more valid than "my @_". - June 2015.

There was no single answer to that clear message. In retrospect I would also don't know what to answer to so much nonsense in one single paragraph.

I concur with rgs that "In my opinion, the more important problem here is the impression that P5P is throwing away without much thought of a perfectly nice and modern language feature (for some value of modern that means "post-FORTRAN"). This could give the impression of a the lack of vision for Perl 5 (and reinforce the "perl is dead" death spiral as perceived by the outside world -- the Perl users)."

Well said, and I knew it beforehand before it became true. But it's pretty easy to foresee such desaster looking at the track record this team is piling up, and how they react to critism, reviews and help. Their latest attempt to come up with an unwritten religious CoC rule, that you need to have faith in the maintainers in order to talk to them is the final sign of "Death by Code of Conduct". It's pretty hard to have faith in someone who decides 90% on the wrong side and only by luck sometimes makes a right decision. Initially I thought this code of conduct could finally stop all the silly bullying and name calling, but it got even worse, and whenever I complain about such abuse p5p punishes the one who complains about abuse, which is clearly only in the interest of the p5p buddies to maintain their powers they are abusing, but not in the interest of anybody else.

Now that p5p decided on this schism and went forward with more totally wrong and outragious decisions to do more and more harm on the perl5 language, I can only recommend to use cperl, the perl5 implementation where such grave mistakes do not happen, where perl5 development actually continues in the spirit before p5p took it over, and where the p5p principles of ruling by incompetence, power and abuse are not tolerated. cperl decisions are made rationally, professionally and most of all "cperl is not a religion". You are allowed to show distrust and criticsm. And in cperl such bugs are actually getting fixed, and the language is not harmed.

You will also hopefully not see those incredibly silly mailinglist threads as the cited one above. It's atrocious. We can only hope that p5p is put to an end soon, and a proper development process can start. There's no way this can go forward like this.

Maybe I should finally mention that fixing the removal of the lexical-topic is not only done by skipping the removal of it. The new CX code e.g. assumes that all the $_ there are global, that no lexical $_ can exist, and fixing this already cost me a few hours when merging with 5.24.0. It's pretty hard to deal with artificially stupidified code. At least the new 5.24 CX code is so wonderful that's it's worth it. +20%

YAML patches done

I've finished most of the needed YAML work, esp. for YAML::XS.

libyaml upstream has now a patch with the new options NonStrict and IndentlessMap https://github.com/yaml/libyaml/pull/8

YAML::XS in https://github.com/ingydotnet/yaml-libyaml-pm/pull/43

YAML::XS writes now proper YAML which can be read with YAML.pm, and passes the CPAN::Meta validation tests. See https://github.com/Perl-Toolchain-Gang/CPAN-Meta/pull/107

For CPAN::Meta I've added validation tests for all existing YAML loaders, so you can see what's going on, and which version is conforming or fails. YAML::XS, YAML::Tiny, CPAN::Meta::YAML, YAML::Syck do pass now, YAML fails.

I have also patches for Parse::CPAN::META and CPAN-Meta-YAML to use the new versions, but only in cperl, no PR yet.

I've also started merging libsyck from upstream into YAML::Syck, which came up with some horrible private extensions, and made them mergable to upstream. But this work is still ongoing at https://github.com/rurban/syck/commits/0.71 and https://github.com/rurban/YAML-Syck/commits/merge-upstream.

TODO (optional)

  • YAML.pm would need a patch to read old YAML::XS with $IndentlessMap=1.
  • YAML::Syck
  • syck

More on YAML, syck looks much better

In my last YAML post I said libsyck is not maintained anymore. I had a look, and this is wrong. Even if _why does not work on it anymore, (he came back btw recently), it is maintained and made some progress in libsyck, which is not reflected in the YAML::Syck perl part.

It is a mess, I admit, but easier fixable than the YAML::XS mess. So I took libsyck upstream, which is at 0.70, and merged it with our changes which are at 0.61. Our perl-specific changes are a complete mess, so I cleaned that up to be acceptable upstream into a new 0.71.

merge back various changes from upstream (my own WIP version 0.71)

  • const char*, recompiled grammar, ...
  • alloc +1 for the final \0
  • add proper type casts

    sanify various unmergable hacks into proper flags, which can be set perl-specific:

  • add emitter->nocomplexkey flag, default=0, 1 for perl.

  • rename syckemit2quoted1 to syckemit1quotedesc
  • and rename scalar2quote1 to scalar1quoteesc (JSON singlequote as single-quoted with dq-like escapes)

    remove some other unmergable hacks:

  • syck_base64enc requires an ending \n

YAML::Syck has many advantaged over YAML::XS. It does support reading and writing to file streams, which means it does not need to slurp each file into a buffer and process that. It can process streamable buffers. libyaml can do that also, but YAML::XS never implemented that. I only added a LoadFile method, but not DumpFile.

YAML::XS doesn't really use the nice architecture libyaml provides, it rather does it's own perl-specific callbacks, bypassing many advantages of libyaml.

libsyck is much better written than libyaml, no question about that. It has much less bugs, much more options to handle, but it got stuck at YAML 1.1 Anybody really needs YAML 1.2? I haven't checked the changes yet.

My changes (still WIP) are at:

So now I'm pondering to convince everybody to ditch YAML and YAML::XS completely in favor of YAML::Syck. Let's see how this will turn out... In fact it's only a tiny patch to CPAN, and I can do that by my own, since CPAN is in core.

My core integration for YAML::XS is at:

What I need now a is good YAML testsuite which merges the validators required by core (CPAN::Meta) and various interop testing as I did with Cpanel::JSON::XS, esp. roundtrips, add the perl module back to syck to give it into sane hands (this migth be tricky as it involves testing with ruby, php, python, ...), do benchmarks and going over the tickets.

What I know is that YAML.pm processing over my cpan prefs is ~10x slower than with YAML::Syck. The current performance is unacceptable, and YAML::XS emitting unindented seq elements for a map child ditto. Maybe I have to fork YAML::XS to a Cpanel::YAML::XS, but most of the fixes need to be done in libyaml itself, and let's see how fixing syck turns out.