The removal of the lexical topic feature in 5.24

Now that p5p removed the broken lexical topic feature in 5.24 it's time to look back how that happened and why it should not have happened, and how cperl fixed that problem.

In 2013 Rafael Garcia-Suarez posted to p5p a plea to Salvaging lexical $_ from deprecation arguing that the recent deprecation of my $_ is wrong, and can easily be fixed.

He explained the problems pretty nicely on the language level, but forgot to explain the internal advantages of a lexical $_ over a global or localized $_. To add that it's pretty simple: A lexical $_ is an indexed slot on the pad array, calculated at compile-time, whilst a local or global $_ is a hash entry in the namespace. It's much faster to lookup, write and restore. The slot index is compiled and carried around in the op. That's why it's called lexical, it is resolved at compile-time, and not at run-time.

The best technical reason why a lexical $_ is needed is given by davem, the only smart guy in the room here at RT #94682. given uses an implicit my $_ internally, and every match uses it.

If you try to read the answers by p5p you will get headaches as their explanations are not only clueless and wrong, they even argue that using $_ lexically is semantically wrong! Eg. rjbs: "I think it is half-baked, confused, and confusing. I don't see how it can be made useful or straightforward, at the language level."p5p

He clearly is confused because of the tricky bugs on the ops he is talking about, but here he was arguing that the semantics are broken(!). Using $_ explicitly or implicitly; or global, local or lexical makes no difference, only to RJBS and to some other porters. Doy for example argues that my $_ should be removed because this example is broken:

my $_ = 'foo'; print = any { $_ eq 'a' } qw(a b c)

which I fixed in 10 minutes by using the lexical $_ in the XS implementation and not the global. See the simple patch at RT #113939

The underlying problems with the implementation of my $_ were quite trivial to fix. I just had to fix the initial check for OPp_TARGLEX in the cperl commit 597e929c9 August 2015, which I published quite well. But the message obviously didn't arrive at the p5p team, because the decided that my $_ cannot be fixed and needs to be removed end of August. This time without any deprecation period.

One other interesting new answer was by zefram that this bug if not fixable, because we'd need support for lexical $_ in every op which supports implicit $_ or takes blocks, such as grep and map, obviously not knowing the codebase that this is exactly what we were doing quite efficiently since 2001. There is e.g. a special lexical grep bit. A git grep GREP_LEX would have helped, which was added 2004 as a special bit to grep.

I think there's no way forward that looks at all similar to what we have. Any lexical topic variable would at least have to be selected on a per-construct basis: you can't have "my $_" causing grep and the like to use a lexical topic. A spearate "grep_lexical" operator should be fine, but if we go that route we should consider forcing the lexical topic variable to have a name other than "$_" to avoid the confusing shadowing. Actual "my $_" should be no more valid than "my @_". - June 2015.

There was no single answer to that clear message. In retrospect I would also don't know what to answer to so much nonsense in one single paragraph.

I concur with rgs that "In my opinion, the more important problem here is the impression that P5P is throwing away without much thought of a perfectly nice and modern language feature (for some value of modern that means "post-FORTRAN"). This could give the impression of a the lack of vision for Perl 5 (and reinforce the "perl is dead" death spiral as perceived by the outside world -- the Perl users)."

Well said, and I knew it beforehand before it became true. But it's pretty easy to foresee such desaster looking at the track record this team is piling up, and how they react to critism, reviews and help. Their latest attempt to come up with an unwritten religious CoC rule, that you need to have faith in the maintainers in order to talk to them is the final sign of "Death by Code of Conduct". It's pretty hard to have faith in someone who decides 90% on the wrong side and only by luck sometimes makes a right decision. Initially I thought this code of conduct could finally stop all the silly bullying and name calling, but it got even worse, and whenever I complain about such abuse p5p punishes the one who complains about abuse, which is clearly only in the interest of the p5p buddies to maintain their powers they are abusing, but not in the interest of anybody else.

Now that p5p decided on this schism and went forward with more totally wrong and outragious decisions to do more and more harm on the perl5 language, I can only recommend to use cperl, the perl5 implementation where such grave mistakes do not happen, where perl5 development actually continues in the spirit before p5p took it over, and where the p5p principles of ruling by incompetence, power and abuse are not tolerated. cperl decisions are made rationally, professionally and most of all "cperl is not a religion". You are allowed to show distrust and criticsm. And in cperl such bugs are actually getting fixed, and the language is not harmed.

You will also hopefully not see those incredibly silly mailinglist threads as the cited one above. It's atrocious. We can only hope that p5p is put to an end soon, and a proper development process can start. There's no way this can go forward like this.

Maybe I should finally mention that fixing the removal of the lexical-topic is not only done by skipping the removal of it. The new CX code e.g. assumes that all the $_ there are global, that no lexical $_ can exist, and fixing this already cost me a few hours when merging with 5.24.0. It's pretty hard to deal with artificially stupidified code. At least the new 5.24 CX code is so wonderful that's it's worth it. +20%

8 Comments

While, yes, ok, 'any' can thereby use the lexical topic, what about pure perl implementations of similar things using &$ signatures? My problems with lexical $_ have never been exactly core-implementation-related so much as "I have so much code that relies on this being dynamically scoped that letting people change that mostly seems like a great way to let the user blow both their feet off".

-- mst

mst: That was the intent of the _ prototype. However given the adoption rate of the lexical-topic feature, it's clear that was not a really wanted language addition...

P5P outsider here.

I read through the whole thread "Salvaging lexical $_ from deprecation" and I think that the intent from RGS was correct: let's just talk more about this issue before ditching something that might be useful indeed. RGS saw value in that construct, and by all means this called for this flag raising.

I didn't see abusing, only discussion on technical merits. Is the reference to the CoC actually referring to some other discussion, or maybe to some moderation applied to that thread? I genuinely don't know.

My perl-guts-fu is too low to judge on the technical discussion, and on the technical preparation of each of the respondents. The impression I got is that even very skillful people like RGS sometimes might not have all the details clear; this applies to many others probably, although I'm not sure everyone is as fair as RGS to admit it and stand corrected. Kudos.

As a user of Perl, I can say that the localization of $_ is likely to break the principle of least surprise in many ways, as many examples showed. Even the introduction of ways to solve things for prototypes made me wonder what would happen for a module that tries to be correct for perl-s before and after the introduction of the new feature (like I was reading regarding Try::Tiny for example).

The matter is also probably rooted as to whether we should abandon Perl 5 at some point in time or not. For good or bad, some design decisions in the past were done and have to be taken into account because otherwise people's code is likely to break. Some degree of innovation can be put in Perl 5, some other probably simply can't without much hoop-jumping and confusing behavior.

I feel the frustration of RGS in the thread and the reference to Perl 7. At a certain point, dead ends in Perl 5 related to past design decisions might make it saner to just declare that a drastic cut from the past is needed. Then it will remain to be seen to what extent this will break the community, how much Perl 7-whatever will be considered a sister language, and all the same.

The _ prototype doesn't really help with things like @$ so far as I can see, unless you're saying that I need to rewrite everything that's currently 'foo { ... } ...' to 'foo sub (_) { ... } ...' which, um, really?

Plus there's plenty of existing code that doesn't call the thing that usess $_ until several layers down, and needing to thread _ prototypes through all of that (and how do you do that when method calls are involved when they don't even honour prototypes?) strikes me as un-perlishly-un-DWIM, and if anything rather reminds me of IO monad lifting in haskell.

I think at this point my POV can be summed up as "lexical $_ was a nice idea in theory, but in practice it seems to've been so baroque to try and use that it causes more hassle than it's worth" - if I've misunderstood anything from the above, I'm open to being corrected, but barring that it just doesn't seem like it worked out, nice though it initially seemed.

The _ prototype failed to be adopted long before the experimental pragma arrived on the scene, much though I guess that doesn't fit the desired narrative.

I find it fascinating that in spite of reini's assertions to me on Twitter earlier that he always focuses on technical aspects first, once again the actual technical questions in this thread have been entirely ignored by reini in favour of further long form personal attacks. One might postulate that this tendency to insult and ignore attempts to engage in technical argument on the merits is a more parsimonious explanation of his exclusion from p5p than some vast conspiracy - readers of this thread should, of course, draw their own conclusions on the basis of the discussion in front of them.

Reini,

You have been told before that personal attacks are not welcome here. I am no longer willing to provide a platform for you to publish them, and as of today, I am therefore disabling your user account here. This is a permanent action; it will be in place for as long as I am directly involved with blogs.perl.org.

To be clear, this is not censorship: there are plenty of other places where people can blog without relying on blogs.perl.org. In addition, I will, if you choose, provide a 301 redirect from your existing posts here to another site. If you wish to take me up on that, please email me the relevant URL on arc@cpan.org.