Perl 7, not quite getting better yet

Hegel remarks somewhere that all great world-historic facts and personages appear, so to speak, twice. He forgot to add: the first time as tragedy, the second time as farce.

The Eighteenth Brumaire of Louis Napoleon - Karl Marx

Sawyer just announced his plans for perl 7. And while Perl 7 sounds like a lovely language, I do see a number of issues:

Cohabitation / Forking

The proposal is presented as a linear progress, I don't believe this is realistic. This would be fork much like the python 3 transition is (which also wanted to be a simple linear progression). As we all know, they're currently in year 12 of a 5 year transition.

There are several problems here. CPAN as an ecosystem is the one that is given most attention to (not without reason; it is without doubt the most important collection of Perl code), but it's not even the biggest problem.

The biggest problem is that /usr/bin/perl is infrastructure. We can't do breaking changes to its basic functionality for the same reason that shell and awk can't. Too many things in too many places are dependent on it, from system administration scripts to bioinformatics workflows to build systems (e.g. autotools, postgresql) and many more.

And this change is vastly breaking. Enabling strict and disabling prototypes (to make way for signatures) will break vast amounts of code, especially in the scripting domain of perl. It's quite telling that 12 years after python3 was released /usr/bin/python isn't a python3 by default on any of the big distributions (Ubuntu, Debian, Fedora, Red Hat, OpenSuse); and arguably python is less entrenched than perl is. I don't believe that /usr/bin/perl will ever be perl7. That means that perl7 can only meaningfully exist if it's set up to coexist alongside of perl5 for a very long time. And that actually comes with a number of challenges that may not seem obvious at first (e.g. colliding script names and man pages).

Releasing a Perl7 will not erase perl5. Perl5 will in all likelihood remain the Perl that's available on any platform regardless of how successful perl7 will be.

Perl 8 and beyond

Major version transitions are costly, and often traumatic (Perl 6 and Python 3 being obvious examples). Communities also take a lot of time catching up with them (again, see above examples); at least a decade if not more.

A big, breaking release is something a mature programming language can only do once per decade or so; anything else would result in two transitions going on at the same time. We shouldn't even be thinking about a perl8 this decade, let alone a perl9. If we are to do a perl7, we must get it right the first time. And I don't think this plan is quite getting it right. And quite frankly, I can't imagine any reason for wanting to do a big breaking release if we'd do 7 right.

We are not ready

The current plan is essentially Enable all non-controversial features by default, and I don't think that that is the best we can do. There are a lot of features that haven't been implemented before because they don't make much sense in a minor release (in particularly the kind that removes syntax like no feature 'indirect'). Releasing it now will force a perl8 relatively soon, and that would be undesirable for all the reasons stated above.

Manpower

We have been failing at shipping non-experimental signatures for more than half a decade now, why would we be able to ship a perl 7? The most significant new feature that made it out of experimental in the past half decade was postfix dereferencing, and while welcome it's not quite a game changer.

Sadly, the most convincing reason not to go through with this may very well be "we may not be able to". I think we need to figure out what problems we can resolve before deciding to actually go forward with this.

Timescale

There's just no way we can do all of the above before the end of the year for a variety of reasons. Not only because it will require adaptations on the perl5 side to enable cohabitation, but also because we will need to sort out a lot of details. Trying to rush is likely to result in a failure, and is not something we can afford. I can't imagine any way of successfully doing this that doesn't involve releasing a v5.34 first (and possibly more).

Then don't upgrade?

The if you don't want your code to break then don't upgrade argument is rather assuming users have a firm control over which perl they are running. This is generally true for million line perl web applications, but this is not true for system perl.

If our objective is to limit ourselves to perlbrew/perlbuild/etc…, many of my objections become moot. But I don't think that should be the target, I think that would exclude a wide range of applications. So no, I don't think saying "then don't upgrade" really solves that problem. We may be able to postpone the problem, but it won't go away by itself.

The absentee/maintainer dichotomy

I do not recognize this distinction at all. Just because I actively maintain my stuff doesn't mean I want to be dealing with other people breaking my code. If I wanted to deal with the whims of a platform breaking trivial things I'd be programming python.

Associating not wanting the language to break with diminishing use of the language is perplexing to me. Perl is a language of which a lot has been written already, and relative to that past popularity not all that much new code is being written. Quite a lot of Perl strongholds are attributable to Perl not breaking, it's uncertain if the pain of this process will be worth the gain.

There's also this suggestion that people who care about backwards compatibility contribute less to the language. This isn't actually explained further but it seems like a rather bold statement to me.

Whom are we serving?

Perl has many different types of users, with many different needs. This is inherent to a language that tries to be useful at 1 line and at 1 million lines.

The argument that has been made in the keynote suggests that the only reason why one would use "old-style Perl" is because you've abandoned your code, and I don't think that is true. Many best practices that are essential when writing large applications are not nearly as valuable in a small script; it would be outright silly to suggest one-liners need strict.

The changes that are proposed are largely serving the manipulexity end of the spectrum. And this is an important user base, but it's not our only user base. For the whipuptitude end of the spectrum, the scripters, this represents their code breaking without them getting anything in return. That is the priority that is being chosen here.

Bad code

I believe this "bad pattern" rhetoric is flawed. Ultimately the only good code is working code, and the only bad code is code that doesn't get the job done. What I hear being described as bad code is actually merely ugly code. And this transition can break stuff for people, and breaking code is bad, whereas ugly code is only a problem to me if it ends up on my plate.

How did we get into this brave new world where one calls judgment on users and deplatforming the ones deemed bad?

This reminds me of a bioinformatician I met at a recent TPC. Was their code strict? No. Did it get the job done? Yes. Why would they care if we in the echo chamber approve of her code, they have more important things to do, like curing ovarian cancer. In my book, they got their priorities straight.

Is this really worth it?

This seems like a lot of pain, just to avoid having to type use v5.32. The real problem of course is that that doesn't only not enable warnings (which we can easily fix for 5.34), but it also doesn't enable signatures (probably the recent feature people care most about). If we can make use v5.34 do those two things, I don't think I need a perl7, even if I understand why some other people feel they do want it. Boilerplate may be annoying, but one line of boilerplate in every file is way more tolerable to me than the pain of a fork.

This was Jesse Vincent's vision 9 years ago, and I still think this is the right trade-off for a platform like Perl.

14 Comments

Thanks for writing that, Leon. Everything you say makes sense ... but I've been wanting for years to have a Perl release that has modern defaults, and there's been resistance to changing things in Perl 5.

I'd be happy to type use v5.34, but I'm wondering if there'd be less resistance to change (and better marketing, easier to understand differences) if that were spelt use v7. Would you be OK with that?

And if there were a binary /usr/bin/perl7 which was like /usr/bin/perl -M7, would that be OK? That is:

  • /usr/bin/perl continues to have Perl 5 semantics by default.
  • To get Perl 7 defaults, either put a v7 in your code, or invoke an interpreter which has that version number in its name.
  • perl -E continues to give you the latest version.

In those circumstances, would it be so bad if Perl 8 followed Perl 7 only a year or two later? Adding a few more files with version numbers in /usr/bin isn't much bloat, and it avoids having to delay Perl 7 until we're done thinking of (and agreeing on) improvements.

Thank you, it very closely reflects my feelings regarding the announcement of Perl 7.

I think we have a solid platform already in cpantesters to support making changes to perl that are already best practices. Obviously more tooling may need to be built on top of it.

Lets be realistic though.

Perl is about to fall off of tiobe. Companies have already ripped it out and banned using it - the few hangers on aren't looking like they will much longer. Distributions like Debian have been aggressively rewriting perl stuff to python etc.

Perl 5 -> 6 was a disaster. Python 2 -> 3 was/is less of a disaster. Ruby breaks by design (and is in a death spiral too) and Java... well what can we say other than that since Java 8 it's license is too risky. Javascript is managing evolving reasonably well considering it is the most widespread language in history.

I think its just a matter of building out the tooling to support it. And as was mentioned, working with distro's to facilitate LTS releases.

@Smylers that sounds like a really sensible approach, but there's some details to be worked out.

If you run a script using perl7 foo.pl, and it "use"s a module, does it apply Perl 7 defaults to the module?

It feels like the answer should be "yes", but it has to be "no" to preserve Perl 5 modules to be usable from Perl 7 -- or you have to invent another syntax to load a module as a Perl 5 module (like Raku's Inline::Perl5 allows "use DBI:from;").

I'd like to push back on the bioinformatician concern. My (professional researcher) opinion is that if a scientist can't be bothered to learn basic software best practices, then they shouldn't be trying to write their own software (and should hire someone else). And if they just borrow software from the internet that is so out-of-date that it isn't maintained or wouldn't work with strict, then (speaking as an active medical journal reviewer) their manuscripts should be promptly rejected and require a code review --- science needs to be confidently reliable, reproducible, and replicable to be trusted.

Scientists are used to paying a lot of money for reagents, instruments that cost tens or hundreds of thousands of dollars, software for hundreds or thousands of dollars, and living wages for grad students and postdocs. So any bioinformatician worth their salt should be able to budget into any grant for development if they need to update or modernize software.

In the early days of the Perl 6 tenure it was viewed as a linear step form perl 5. Over weeks and months and years that changed. Lets not let the same thing happen to Perl 7. Even if all that happens is a major version bump and all other release planning remains constant it'll be a strong message to the community. Lots of us were not yet in the industry when the change was made from perl 4.x to 5.x. Back then distros took years before "system perl" was advanced from 4 to 5. If perl was even distributed in the system. Eventually it did happen.

Fossils know how to preserve themselves. Geology and archaeology are noble sciences. The living ecosystem does not need to over concern itself with biologically stable samples.

Moritz, I wondered who was going to bring that up! Yep, it needs some detail, but I think it could be made to work.

  • As you say, under use v7 semantics, if use loads a module and lets it run as v7, then we'd need use5 or some other keyword for loading an old module that doesn't know about Perl 7.
  • Or, if somebody thinks of a synonym for ‘use’, maybe use itself always defaults to v5 (the module itself can always opt in to v7 semantics with a use v7) and the new keyword means to load a module keeping the semantics of the surrounding program.
  • Or, maybe its ‘package’ that needs a synonym? If a module file starts with, say, module Foo::Bar (instead of package Foo::Bar) then it indicates new semantics.

But the problem with new synonyms is we only get to do that once. So a new keyword can only either mean ‘use v7 semantics’ (which doesn't give you v8 improvements when those come along), or it means ‘use the caller's semantics’ (or the latest semantics), which could cause problems if a v7 module unexpectedly gets v8 semantics applied. So, ugly as it is, maybe we just need package7 keyword, thereby allowing for package8 and so on.

The key tension here is that there are two main scenarios, which would benefit from exactly opposite behaviour:

  • If I'm writing a some software with multiple modules in Perl 7, having specified Perl 7 for my main program (by whatever means), it's tedious boilerplate to have to re-specify that separately for every single module file. And can lead to surprising behaviour and hard-to-find bugs if there's just one file somewhere I forget it.
  • If I'm using a Cpan module written by somebody else, I just want it to work. Unless it self-identifies as being for Perl 7, it should run as Perl 5.

But typically I'll be doing both of those in the same file, and the command for loading other parts of my software and for loading external Cpan modules is exactly the same, so Perl can't tell the difference. (Well, maybe it sometimes could guess, using heuristics based on the path. But that sounds brittle and like it introduce more pain than it would avoid.)

So maybe actually the second approach above, of two different use statements, maps best to that: use itself for ‘load this external library’, which is conservative and resets to v5 semantics (until the module itself request otherwise); and a new keyword which means ‘load another component of this project’, which propagates the current version into the loaded module.

Anybody using v7 semantics on Cpan would have to declare that — but that's fair enough on a public repository to state your minimum version number.

Smylers, another nugget to consider: Raku's "use DBI:from;" only work because Raku and Perl 5 have different library search paths (and methods, but that's secondary).

If you plan to use the same search paths for mixed Perl 5 + 7 libraries, then the information about which language to use has to be in the module, not in the caller.

Otherwise you run into weird issues where you have "use5 A::B" but "use7 A::C", and then A::B cannot upgrade to Perl 7, because all the callers still use "use5" to load it (whatever the syntax turns out to be). Also, you spread the knowledge what language something is implemented in all around the place.

So, I guess it was kinda short-sighted for me to bring up the "use DBI:from;" example, because it doesn't map well onto the perl 5 + 7 situation.

Now my personal favorite is "use v7;", because it has a clear scope, doesn't introduce a flurry keywords for future versions, and has an obvious upgrade path to Perl 8.

you run into weird issues where you have "use5 A::B" but "use7 A::C", and then A::B cannot upgrade to Perl 7, because all the callers still use "use5" to load it

Not quite. use wouldn't mean ‘load and enforce v5 semantics regardless of what the file says’, but ‘load with v5 semantics by default’. A::B could still put a use v7 in there to switch itself to modern meanings. The point would be to avoid inadvertently loading a legacy module with modern semantics that it didn't know about, but nothing stops it opting in to them if it does know about them.

PS: Apologies to Leon for taking over his blog. I really should find the time to watch Sawyer's video and then reply on P5P (P7P?).

Can we just have a /usr/bin/perl-ng (for 'next gen', or any other cheesy suffix we can think of)? It doesn't take over /usr/bin/perl, we won't end up with multiple perl8, perl9, etc... And it still has "perl" in it. /usr/bin/perl-ng will be 7 and beyond, /usr/bin/perl will be 5.x. Distros can ship a perl 7+, and a perl 5.x. It will be annoying to remember adding -ng to perldoc, cpan, and a few others, but, seems like something we can get used to (I'd just use shell aliases or something).

Nice article, I see most of your points. For the purposes of accuracy, however, you might like to correct what you say about Python3 and the Linux distros. Fedora (since Fedora 32 which is the latest stable) has switched to Python3 by default (and yes, this has upset many dev folk that were still relying on Python2). Gentoo (I think) has a compilable option to switch to Python3 and OpenMandriva has also ditched Python2. So, things are happening there.

Leave a comment

About Leon Timmermans

user-pic