Perl7 is a fork of values

Before reading this, you should watch this video where Bryan Cantrill explains a value-conflict between Joyent and Node.js, I believe we have a similar problem.

In it he defines a list of project values:

values

All these values are important - but they are in tension. In the end one has to choose between them.

Perl's has traditionally prioritized certain values over these others, and in my experience these are:

  • Expressiveness
  • Extensibility
  • Stability

Perl 7, not quite getting better yet

Hegel remarks somewhere that all great world-historic facts and personages appear, so to speak, twice. He forgot to add: the first time as tragedy, the second time as farce.

The Eighteenth Brumaire of Louis Napoleon - Karl Marx

Sawyer just announced his plans for perl 7. And while Perl 7 sounds like a lovely language, I do see a number of issues:

Cohabitation / Forking

The proposal is presented as a linear progress, I don't believe this is realistic. This would be fork much like the python 3 transition is (which also wanted to be a simple linear progression). As we all know, they're currently in year 12 of a 5 year transition.

There are several problems here. CPAN as an ecosystem is the one that is given most attention to (not without reason; it is without doubt the most important collection of Perl code), but it's not even the biggest problem.

The biggest problem is that /usr/bin/perl is infrastructure. We can't do breaking changes to its basic functionality for the same reason that shell and awk can't. Too many things in too many places are dependent on it, from system administration scripts to bioinformatics workflows to build systems (e.g. autotools, postgresql) and many more.

And this change is vastly breaking. Enabling strict and disabling prototypes (to make way for signatures) will break vast amounts of code, especially in the scripting domain of perl. It's quite telling that 12 years after python3 was released /usr/bin/python isn't a python3 by default on any of the big distributions (Ubuntu, Debian, Fedora, Red Hat, OpenSuse); and arguably python is less entrenched than perl is. I don't believe that /usr/bin/perl will ever be perl7. That means that perl7 can only meaningfully exist if it's set up to coexist alongside of perl5 for a very long time. And that actually comes with a number of challenges that may not seem obvious at first (e.g. colliding script names and man pages).

Releasing a Perl7 will not erase perl5. Perl5 will in all likelihood remain the Perl that's available on any platform regardless of how successful perl7 will be.

Perl 8 and beyond

Major version transitions are costly, and often traumatic (Perl 6 and Python 3 being obvious examples). Communities also take a lot of time catching up with them (again, see above examples); at least a decade if not more.

A big, breaking release is something a mature programming language can only do once per decade or so; anything else would result in two transitions going on at the same time. We shouldn't even be thinking about a perl8 this decade, let alone a perl9. If we are to do a perl7, we must get it right the first time. And I don't think this plan is quite getting it right. And quite frankly, I can't imagine any reason for wanting to do a big breaking release if we'd do 7 right.

We are not ready

The current plan is essentially Enable all non-controversial features by default, and I don't think that that is the best we can do. There are a lot of features that haven't been implemented before because they don't make much sense in a minor release (in particularly the kind that removes syntax like no feature 'indirect'). Releasing it now will force a perl8 relatively soon, and that would be undesirable for all the reasons stated above.

Manpower

We have been failing at shipping non-experimental signatures for more than half a decade now, why would we be able to ship a perl 7? The most significant new feature that made it out of experimental in the past half decade was postfix dereferencing, and while welcome it's not quite a game changer.

Sadly, the most convincing reason not to go through with this may very well be "we may not be able to". I think we need to figure out what problems we can resolve before deciding to actually go forward with this.

Timescale

There's just no way we can do all of the above before the end of the year for a variety of reasons. Not only because it will require adaptations on the perl5 side to enable cohabitation, but also because we will need to sort out a lot of details. Trying to rush is likely to result in a failure, and is not something we can afford. I can't imagine any way of successfully doing this that doesn't involve releasing a v5.34 first (and possibly more).

Then don't upgrade?

The if you don't want your code to break then don't upgrade argument is rather assuming users have a firm control over which perl they are running. This is generally true for million line perl web applications, but this is not true for system perl.

If our objective is to limit ourselves to perlbrew/perlbuild/etc…, many of my objections become moot. But I don't think that should be the target, I think that would exclude a wide range of applications. So no, I don't think saying "then don't upgrade" really solves that problem. We may be able to postpone the problem, but it won't go away by itself.

The absentee/maintainer dichotomy

I do not recognize this distinction at all. Just because I actively maintain my stuff doesn't mean I want to be dealing with other people breaking my code. If I wanted to deal with the whims of a platform breaking trivial things I'd be programming python.

Associating not wanting the language to break with diminishing use of the language is perplexing to me. Perl is a language of which a lot has been written already, and relative to that past popularity not all that much new code is being written. Quite a lot of Perl strongholds are attributable to Perl not breaking, it's uncertain if the pain of this process will be worth the gain.

There's also this suggestion that people who care about backwards compatibility contribute less to the language. This isn't actually explained further but it seems like a rather bold statement to me.

Whom are we serving?

Perl has many different types of users, with many different needs. This is inherent to a language that tries to be useful at 1 line and at 1 million lines.

The argument that has been made in the keynote suggests that the only reason why one would use "old-style Perl" is because you've abandoned your code, and I don't think that is true. Many best practices that are essential when writing large applications are not nearly as valuable in a small script; it would be outright silly to suggest one-liners need strict.

The changes that are proposed are largely serving the manipulexity end of the spectrum. And this is an important user base, but it's not our only user base. For the whipuptitude end of the spectrum, the scripters, this represents their code breaking without them getting anything in return. That is the priority that is being chosen here.

Bad code

I believe this "bad pattern" rhetoric is flawed. Ultimately the only good code is working code, and the only bad code is code that doesn't get the job done. What I hear being described as bad code is actually merely ugly code. And this transition can break stuff for people, and breaking code is bad, whereas ugly code is only a problem to me if it ends up on my plate.

How did we get into this brave new world where one calls judgment on users and deplatforming the ones deemed bad?

This reminds me of a bioinformatician I met at a recent TPC. Was their code strict? No. Did it get the job done? Yes. Why would they care if we in the echo chamber approve of her code, they have more important things to do, like curing ovarian cancer. In my book, they got their priorities straight.

Is this really worth it?

This seems like a lot of pain, just to avoid having to type use v5.32. The real problem of course is that that doesn't only not enable warnings (which we can easily fix for 5.34), but it also doesn't enable signatures (probably the recent feature people care most about). If we can make use v5.34 do those two things, I don't think I need a perl7, even if I understand why some other people feel they do want it. Boilerplate may be annoying, but one line of boilerplate in every file is way more tolerable to me than the pain of a fork.

This was Jesse Vincent's vision 9 years ago, and I still think this is the right trade-off for a platform like Perl.

Smartmatch in 5.27.7

What happened?

In the latest development release of perl, smartmatch changed quite a bit.

Almost everything you believed about smartmatching is now wrong

No really, everything. All previous rules are gone except a single one: you can smartmatch against any object that overloads smartmatching (the only "objects" that overload them out of the box are qr// regexps).

Matching against a scalar value? Gone. Matching against a list of values? Also gone.

when is no more.

The when keyword is gone, split into two keywords: whereis and whereso; one smartmatches the value against the current subject and the other does a simple boolean check much like if. I'll let you guess which is which. This split is for good reasons (when sometimes does one and sometimes the other, sometimes depending on things like optimizations), but that doesn't make this any more intuitive.

use 5.010/use 5.028 won't guard you from this.

It would have been possible to support both behaviors, because the old behavior is already using feature.pm. In fact one could even enable old and new style when at the same time in a scope without problems. None of this was done though.

My suggestions

new smartmatch should be more useful.

Right now one can't do anything with it without a helper library (like my Smart::Match). That's just silly.

The insanity of old style matching was that the overloads depended on both operands, this gave rise to hard to predict behavior, but that doesn't mean one can't define useful behavior that only depends on the right side that follows the most common use-cases. In particular matching scalars stringwise, and making $foo ~~ ["bar", "baz" ] mean $foo in [ "bar", "baz" ].

This should be opt-in

Despite retroactively adding warnings the feature is experimental, it has become a widely used feature. This change is breaking a (yet unknown but significant) number of CPAN modules, and likely much more code in darkpan. Breaking this unless strictly necessary is dumb.

And it isn't necessary. We can easily only enable the new behavior when asked for. That way we can improve smartmatching without breaking a decade worth of smartmatching code.

We need better words

whereso and whereis are way too confusing. I'm not sure what that would look like but this just doesn't cut it.

Above all, we need a better process

Somehow, p5p made a fundamental breaking change to the language without even trying to involve the wider community. This blogpost shouldn't have been the first time the wider community hears about it. And we need that wider community IMHO because no one on p5p (myself included) has the kind of language design talent that's required to do the sort of thing we did here. I don't know what the solution would look like exactly, but I like this process even less than I like the outcome so far. I must admit I'm somewhat jealous of Python's PEP process, though I'm not sure that would work without a language designer to guide it.

File::Slurp is broken and wrong

If you are using File::Slurp, you should possibly reconsider. Basically, there are three reasons to do so;

It is wrong in a lot of cases.

File::Slurp predates IO layers, and as such doesn't take them into account well. A few years ago, after some complaints, an attempt was done to make it handle encodings. This was nothing short of being wrong.

The best known bug in this area is #83126, which means that :encoding() layers are always interpreted as :utf8. This not only means that UTF-8 encoded text is not validated (which can be a security risk), but also that files in other encodings (such as UTF-16) will be read as UTF-8, which surely will give an incorrect result.

Likewise it's not handling :crlf correctly, in particular explicitly asking for :crlf will always disable it, even on Windows.

Basically, it's doing all binmodes wrong except the one you shouldn't be using anyway (:utf8), and you should pretty much always be using a binmode, so there's no way to win really.

The interface is poorly huffmanized.

Huffmanization is the process of making commonly used operations shorter. File::Slurp is failing to huffmanize in the unicode world of 2015. Text files are usually UTF-8 nowadays, which in File::Slurp would typically be read_file($filename, binmode => ':raw:utf8'). The shortest option, read_file($filename), does something that most people don't really want anymore: latin-1 encoded files with platforms specific line-endings.

This is mainly the fault of perl itself (backwards compatibility is a PITA), but a library can work around this to make the programmers life easier.

It is poorly maintained

The critical bug mentioned above has been known for about two years, yet the author hasn't even bothered to respond to it, let alone fix it. There hasn't been a release in 4 years despite an increasingly long list of issues. Worst yet, this isn't the first time such a thing happens; before his last maintenance surge in the spring of 2011 the author was also missing-in-action for years. This negligence is inexcusable for a module that is so commonly depended upon.

Recommendations

Instead of File::Slurp, I recommend you use one of these modules depending on your needs:

If your needs are minimal, I'd recommend my File::Slurper. It provides correct, fast and easy to use slurping and spewing functions.

If your needs are average (which is the case for most people), I'd recommend Path::Tiny. This provides a well-balanced set of functions for dealing with file paths and contents.

If you want to go for maximal overkill, try IO::All. It will do everything you can imagine and more.

2 weeks of perl

It all started with the Cluj.pm summer meeting on the 9th of August. I happened to be around there, so popped in. Cluj.pm is a refreshingly young perl monger group (I might even have been older than the average age there, that's a first for me). At first I didn't know anyone, other than the guest speaker Mark Keating, but after my presentation I had lots of people approaching me and I had a brilliant evening.

A short week later I flew to Germany, for the Perl Reunification Summit in Perl. Like Schwern I arrived a day earlier than most, so I had a calm start of the meetup. It was mostly a gathering of familiar to me faces, though a significant number I hadn't really spoken to before, specially the Perl 6 guys, -Ofun attracts awesome people. I spent most of the PRS talking to people, and doing a little coding (both related and unrelated). It was a very enlightening meetup.

Lastly, there was YAPC::EU. Despite the sometimes unbearable heat, it was awesome. At some points it seemed a bit less organized than my previous YAPCs, but that may also be me noticing more of what's going on. I spent most of my time in the hallway track, which extended into the pub track, and I spent enough time discussing (and occasionally ranting) that it's a miracle that I still have voice left. In between I found enough time to attend some talks, interestingly I attended most of them on the day I gave one myself. After doing threads last year I could only top it with signals this year. I'll have a challenge to come up with a crazier, I think I'll have to look in a vastly different direction (I have ideas already). After a full week of conferencing, I was relieved to be going home though.

So in all I met Mark Keating in 3 different places in 2 weeks time, I'd almost accuse him of stalking me!