Fixed 5.22 problems during my compiler port

I uncovered and fixed many 5.22 problems with cperl already, but in the last months I was busy to port the 3 compilers B::C, B::CC and B::Bytecode to 5.22.

As I said in my interview it's my belief that if all current p5p core committers would stop committing their bad code it would be actually be the best for the perl5 project. They weren't able to implemented any of the already properly designed features from perl6 in the last 12 years, and every feature they did implement is just so horrifibly bad, making our already bad code base, which led to reimplementation efforts of perl6/parrot with a better core, even worse. With cperl I can only undo a little, but when they start breaking the API and planned features in an incompatible way they should just stop.

Nevertheless, 5.22 added a significant improvement from outside, syber's monomorphic inline caching for method calls besides the internal improvement of multideref by Dave Mitchell.

Now to the problems I had to fix in the last months with that 5.22.0 release:

1. Father broke ByteLoader

cperl #75 perl-compiler

This is something I cannot fix in the compiler. I updated my perl patcher App::perlall with new --patches=Compiler patches to fix this, and cperl of course also has this fix.

I had to write a complicated probe mechanism for ByteLoader to check if the used perl5.22 version is already patched or not. Probing a to-be-built XS submodule is not that easy. A typical chicken and egg problem. I could use my already existing B::C::Flags helper config, which allows custom compiler settings. There I initialize the variable $B::C::Flags::have_byteloader with undef, and when the XS modules are all built I call a helper script to probe for a working ByteLoader, and patch $B::C::Flags::have_byteloader to 0 or 1. I can use this then in the tests to skip or run the bytecode tests. And I had to put this helper script into the hints directory to skip it from being installed. Messing with EUMM libscan() was too dirty for me.

The internal compiler op.c creates a new main or eval environment with newPROG(), setting the entry points PL_main_start and PL_main_root from the intermediate parsed PL_compcv. In the case of en empty source the parser always adds a final ; semicolon, which leads to an empty optree starting with OP_STUB.

But with commit 34b5495 for [perl #77452] the compiler now always adds a LINESEQ in front of the STUB, but the logic in newPROG for source filters which already setup PL_main_start and PL_main_root wasnot changed, which led to a broken ByteLoader.

This is an interesting commit as it added a lot of wrong comments about the inner working of this, but didn't update the logic.

The fix in cperl is here and for perlall here, and my perlbug report did not get through.

I can only guess that p5p blocked me again, because they didn't like me to call them incompetent. Blocking bug reports and fixes is worse than just incompetence, but I got used to that recently. They blocked my simple fix for the horrific double-readonly system, and they proudly announced last week some new optimization regarding faster arithmetic, but didn't have a look into my fast arithmetic optimizations which I wrote half a year ago, and which makes them look very bad in the end. Everybody applauded poor Dave for this "fantastic breakthrough". The guys are really that simple. Looking through my improvements would have wasted less time and would have improved it upstream by 30% not just 10%.

2. Dave couldn't implemented multideref access for the compiler

cperl #76 perl-compiler #341

Multideref merges sequential hash or array access into one compressed op. This is a pretty good compiler optimization, if the B design would not be so bad.

The upstream design of the new 5.22 B::UNOP_AUX::aux_list method deviates significantly from proper B design. aux_list requires the curcv to be provided, which is not trivial to do for a B module, and it needs this to resolve shared SVs beforehand. Requiring the curcv to resolve the padoffset is unneeded and does not help B and any of its clients. Clients need the padoffset and resolving it e.g. in B::Deparse is to be done in B as with all other threaded and shared SV accessing methods.

Thanksfully I can patch most of B bugs by myself, and don't have to fork it publicly into a worse name. B is already a good enough name, and I don't want to deviate from that, even if p5p consistently refused to maintain B properly in the last years. There was some short time a few years ago where I could work without a patched B, but this period only lasted very shortly, and none of my fixes were applied, while other new horrific mistakes made it in.

3. Missing HV::ENAMES api

perl-compiler commit ++

Stashes can be aliased to seperate namespaces, and the ENAMES API to access this names never made it into B, and thus never into a compiler. Namespaces aliases are rather seldom, so it caused not too much trouble, but now I added ENAMES and could hereby fix most of the remaining compiler limitations, even for 5.14.

4. Missing PADNAME B api

I explained that technically in my interview. Currently we limit the max name length of lexical variables to 60, because we statically allocate the buffers for them. It is not a practical problem, and I'll optimize that sooner or later to smaller static structs.

5. Fixed HEK assertions

HEK's (shared hash keys) are still dynamic, not static, but I could fix the remaining refcount issues at least.

The cperl code to support static HEKs is already there, but I still need to add compiler code and probes to support that.

6. Broken B::RV->FLAGS for GVOP_gv -> CVREF

5.22 has a wrong RV->FLAGS for a GVOP_gv pointing to a CVREF gv(cv ref:). It returns the flags for a GV (0x808009) where it should be just 0x801, a ROK RV. This is suddenly broken in 5.22 because it's a new optimization they did, and of course wrong.

cperl #63

I haven't fixed that yet in cperl, I just a workaround in the compiler with this patch


Overall we are very happy with the new 5.22 compiler, though we are not yet using the much more advanced cperl optimizations. The B::C optimizations alone lead to ~20% less memory, with cperl and its compiled readonly hashes for Config and warnings and its upcoming support for static GV/AV/CV/PAD/HEK layout it's much more dramatic. This will be a real COW (copy-on-write) mechanism then, being able to statically allocate readonly buffers, and copy it to the heap, when it's being changed. For the compiler we only need to ensure that static buffers are not freed, which is trivial with the added flag.

-m support for perlcc, compiling to modules, not single binaries is also improving. This can split various optimizations per module/.pm file, so we can use B::CC compiled modules or even rperl compiled modules, compile-times should go down from 20min to ~5min, with much faster smoker feedbacks, and pushing updates live is much faster, because they will be much smaller. The old compile times were 2 hours.

But since fixing B::C for 5.22 needed so much more time than expected I couldn't add most of the planned cperl optimizations for the upcoming cperl-5.22.2 release and B-C-1.53 release.

8 Comments

Can we try cperl via perlbrew/plenv?

Also, publicly stating "John broke xyz" probably doesnt help win people over.

To install cperl using plenv you need to first install IO::Socket::SSL. For example using cpanm:
cpanm -n IO::Socket::SSL
Then install cperl with:
plenv install -j 4 -Dusedevel -Dusecperl --as cperl-5.22.1 https://github.com/perl11/cperl/archive/cperl-5.22.1.tar.gz

I wrote small script to automate most of it. Available at my bitbucket repository.

I think Perl::Build has support:

https://metacpan.org/changes/distribution/Perl-Build#L5

but haven't test it yet.

I wish somehow we could have a Perl development process that reflected what the vast majority of Perl programmers needed from a language, but I guess that never happening :(

> Also, publicly stating "John broke xyz" probably doesnt help win people over.

Yes, that.

I agree with you having incompetent coworkers/whatever-fellow-open-source-developers-are-called is a big problem. As someone on the outside looking in, this situation really bugs me. Your work on cperl seems pretty awesome, but cperl would be even more awesome if it didn't exist, and instead you actually cooperated with p5p to make perl itself more awesome. Just as you've chosen to point out their incompetence; instead, you could choose to bury the hatchet and cooperate with them. Replace personal attacks with
technical discussions.

p5p has spoken: by banning you repeatedly for "pointing out others incompetence" or as they see it "...following uncivilized behavior displayed on the list. Ricardo Signes asked that Reini focus on technical discussions and apologize for the comments made, or resign from the list. following a refusal to comply, Reini was banned." (From one of Sawyer X's p5p summaries on blogs.perl.org.) To me that sounds like you've been banned permanently instead of just for one month like you have in the past. To me this sounds like p5p has had it with you "pointing out others incompetence", and banned you permanently. Moreover, p5p has just told you that your plan to bully the incompetent members of p5p into quitting as you mention you did with parrot has failed.

Since your plan has failed, because p5p won't tolerate their incompetence being pointed out (as you see it). You should do as they asked above to get unpermabanned, and just apologize for the comments you made, and promise not to make similar comments in the future. I believe the problem is not that the others on p5p are incompetent; instead, I think you're just a better developer than they are. A good example is your arithmetic optimizations being better than Dave's. I don't follow p5p, but Dave seems to be one of the best developers on the list, and if you're better than he is, then you could be the best one on the list. I'm not saying you're the best for no reason; instead, I'm saying it to make a cool point below... (Don't get too hung up on this "you're the best programmer" crap. It's just a device to make the point below.)

Imagine p5p was a software development company instead of an open source project mailing list. Since you're the best programmer, you're job title would be "Programmer 4". The better p5p developers would be "Programmer 3"s. The average programmers would be "Programmer 2"s, and the new hires would be "Programmer 1"s.

If you as "Programmer 4" call other programmers "incompetent" just because they aren't as good as you are. Then they complain to your boss or HR, and they get you suspended (banned for a month), and after your misbehavior continues then they get you fired (banned permanently). That's old history, and has already happened. Now you need to apologize for the comments you've made to get rehired (unbanned).

Instead, if you as "Programmer 4" realize that you're a better programmer than everyone else in the company (on p5p), you would realize that if someone commits what you consider to be bad code. You would realize that they did not do this intentionally (because who would?); instead, the person who commited bad code didn't even know they commited bad code, and it's your job as "Programmer 4" to mentor them, and show them not tell them that their code is bad, and why their code is bad, and how to make their code better in the future.

This was where I was going with that "you're a better programmer than they are crap". Since, you're a better programmer than they are, you're a whole level higher than they are, and until you realize it you'll never be able to communicate with them as equals. You see the problems in their code, but they don't see them, so when you just call them "incompetent" or tell them their code is "bad", they don't know why you're doing it, get pissed off, and ban you.

Anyway, I think p5p has gotten tired of your behavior, and if you attempt to continue it, they'll just ban you again except that I think you've been banned permanently, so there's no need to ban you again. So, acknowledge your failure to get them to quit, and instead bury the hatchet, and work with p5p to make perl better.

[I wrote this, because I really like your blog posts especially how you effortlessly explain perl's internals like you actually understand how it actually works. And perl would be even better if you'd work with the other corehackers on p5p, for example, if you and Dave combined your arithmetic optimizations, perl could be even faster than either's patches alone.]

++dly Thanks for the good description of mentoring vs. badgering. Alas, transferring skills is another layer of talent and effort on top of employing those skills. At the moment this split seems healthy for all, though I share your hope that these projects can eventually successfully combine forces.