Grinnz

I've written a new blog post on Perl 7 (prev: Perl 7: A Risk-Benefit Analysis and Perl 7 By Default). You can find it, and likely my future posts, on dev.to#perl, for similar reasons as mentioned here.

Perl 7: A Modest Proposal

Perl 7 has been announced as the next direction of Perl development. My previous blog post explored at a high level the risks and benefits of the announced direction, as well as those of a more incremental proposal. The primary and critical difference between these two approaches is the decision to change interpreter defaults in an incompatible manner; specifically, to have strict and warnings and possibly other features enabled by default for code that does not specify otherwise. I would like to explore each of the arguments presented for this design choice.

Optimizing For New

The primary benefit of changing the implicit defaults is, of course, to allow Perl programmers to write code in a more modern way and newcomers to program in a safer environment without having to know the sometimes arcane or niche ways to activate such an environment.

In a perfect world, a change in defaults helps with this problem. But in practice, it will be a step backwards. As it stands today, people continue to use Perl tutorials and code examples from the early 2000s, and this won't change because some new ones will have a 7 attached. The oldest examples out there will fail to run under this Perl 7 (in the best case; the worst case is more unpredictable), which is most likely to turn the newcomer away. These aren't great examples to learn Perl from in any case, but the chance of continued engagement is better than none. The vast majority of tutorials and examples will still recommend and include strict and warnings, which will behave no differently under this Perl 7.

And finally, any new material written for this Perl 7 will omit the boilerplate entirely. Since it will have no boilerplate, it will have no indication that the code was written to a Perl 7 standard, or preventing it from running on a Perl 5 interpreter, as these features do not carry any inherent incompatibility. This puts the onus on every person sharing such code to include these caveats, with no guarantee that a reader will understand such context. Even if only the bare minimum of strict and warnings are enabled by default, this code run on a Perl 5 interpreter will silently miss the compile- and run-time safety features these pragmas provide, resulting in less exposure of the modern coding environment we wish to present, not more. If other features such as say are enabled by default in Perl 7, such code on a Perl 5 interpreter will result in the same confusing errors one currently gets when omitting the necessary feature.

Temporarily accepting the optimistic assumption that changing the programming environment based on Perl version rather than explicit declaration will be more attractive to new programmers; we should not ignore the needs of the existing userbase. Yes, Perl use is declining; no, it is not dead (as evidenced by having this discussion). There are active CPAN authors, active use of Perl in corporate codebases, 15 year old oneliners and scripts used for systems administration. New defaults do not inherently improve the experience of any of these users; they have not only existing code that may stop working, but existing expertise, code style and assumptions that affect new code they write. New defaults require re-evaluation of both of these aspects; a new version declaration requires only re-evaluating what is written or converted within that scope.

This is not a choice to optimize for new users instead of abandoned code, but a choice to optimize for appearance instead of both existing code and active users.

Why Not Both

As changing the defaults would necessarily alienate those who wish to continue using old code without having to rewrite it, it has been suggested that such users can simply continue to use Perl 5, that LTS versions of Perl 5 would be provided for many years, and further that distributors of Perl are likely to distribute separate perl5 and perl7 binaries.

If we expect Perl 7 to be provided ultimately as a separate incompatible binary, then we are expecting a fork of the Perl ecosystem. Existing code will not be guaranteed to run on the new Perl 7 binary, and new Perl 7 code will at best fail to run on the old Perl 5 binaries, and at worst silently fail to inform the user of errors. Code analysis tools like perlcritic will need to be told which Perl they are analyzing as the code will not contain any hints itself. Any discussion of Perl coding will have to start with "Are you using Perl 5 or Perl 7?"

This approach is technically feasible with an uncomfortable transition period, and would make sense if Perl 7 was to be a significant evolution of the language, and brought great benefits inherent in the incompatibility. This is simply not the case; all of the foreseeable benefits can be realized without breaking changes, and thus this design and its potential for confusion and schism is unnecessary.

The greatest risk of this approach, of course, is the scenario that the Perl 7 interpreter never reaches widespread adoption and development restarts on Perl 5. This risk should not be taken lightly, particularly as this approach seems predicated on the expectation that most people will in fact stick with Perl 5 if Perl 7 is not compatible. We should certainly learn from the Raku saga, and not follow the same path without even a new language to offer this time.

A New Culture

Since the new defaults will not contain any revolutionary new feature or ability, a Perl 7 with changed defaults would not provide sufficient value inherent in the incompatibility. This has been rationalized with a secondary goal to change the culture of Perl development. The current Perl policy is one that it has held for decades: backwards compatibility is paramount, despite the inconvenience. It is suggested that future Perl releases will in fact follow the Perl 7 model to a greater extent, and thus the Perl 7 changes to defaults are designed to set a precedent of major versions breaking compatibility, possibly without the restrained nature of the currently suggested changes.

This seems short-sighted. By all measures, all of the features that could be introduced in future major versions will similarly be easy to introduce in a non-breaking manner. And more importantly, do we really want to go down the road of releasing major versions that break code as a policy rather than when it is necessary and beneficial? This approach does not seem to be in line with Perl's defining pragmatism. Perl is a known quantity, for better or worse; we cannot reinvent what Perl means to the world, only how we present its reality.

They Did It

Python and other languages have experienced major versions with significant breaking changes. After a long and arduous transition period, the Python community is just now beginning to coalesce around Python 3. This appears to show that major versions can break compatibility and the end result can justify the difficult transition.

But the comparison is flawed. The Python 2 to 3 change primarily heralded a one-time incompatible change of explicitly marking strings as bytes or characters, resulting in an objectively better language feature that can only work when applied throughout the whole stack of code. Perl has no feature that even approaches implementing an explicit distinction like Python added, and all of Perl's existing features are easily able to be limited to a lexical scope and leave other code unmodified. Thus, Perl has no feature that would benefit from the significant risk of this approach in the way that Python did. And far from the idea that it facilitates a culture of breaking changes in major versions, the ordeal of Python 3 has very likely made developers more apprehensive of future breaking versions so as not to repeat the trauma without good reason.

Moving Forward

Another suggested impetus for incompatible major versions is to allow Perl development to continue more smoothly, without the shackles of past cruft.

It is of course a worthwhile goal, but changing interpreter defaults does not assist in this goal. Changing defaults does not affect policy or practical ability to remove or alter actual language features. Of course the established policy and application of it can be debated, and the ability to leverage major versions provides significant opportunity in this regard, but this discussion is unrelated to the choice to change user-facing defaults.

Further, it has been pointed out by two of the three people that have recently been working on forward-moving language features that user-facing features and syntax are not a significant barrier to such development. See these messages from Dave Mitchell, who has been working on the signatures feature, and Paul Evans, who recently implemented the isa feature and has many more planned including core try/catch. (Zefram, who recently implemented chained comparisons, has not weighed in on the discussion to my knowledge.)

Conclusion

Moving forward in Perl development and culture would be best achieved by the simple and consistent approach of leveraging the use v7 declaration to provide a modern environment lexically, and interpreting code without such a declaration the same way as now in Perl 5.

The major version provides significant opportunity for messaging of new features, as well as what features such a declaration will enable. The announcement of Perl 7 can detail all of the great things you get in a scope with the line use v7. It is significantly easier to remember that, hypothetically, use v7 will enable warnings and use v8 will enable signatures, than to remember that use v5.12 enables strict and use v5.16 enables __SUB__. The declaration provides explicit reference that the code within that file or code snippet requires that version of Perl, without relying on context of shebangs, filenames, or surrounding text that can get left out. And it provides an explicit hint to static parsers (syntax highlighters, perlcritic, etc) how the following code should be interpreted. Someday, when we are ready for a Perl 8, it will not require repeating any of the plans discussed above for introducing incremental incompatibility; it will only require use v8.

I believe making good use of a new major version is extremely important to portraying the continued and forward development of Perl to the wider programming community. A major version with major features can be a significant boon to jumpstart the stagnating perception of Perl and bring it in line with the reality of its development. But changing the interpreter defaults is unnecessary, irresponsible, and counterproductive. A simple scoped and versioned declaration to enable the modern coding environment provides a stronger way forward and no new risk. It allows Perl 5 users and Perl 7 users to just be Perl users. It should be our default choice.

Reddit comments

At the recent Conference in the Cloud for Perl and Raku, Sawyer X (the pumpking of Perl) announced an ambitious plan for Perl 7. Since Perl 6 was renamed to Raku to better communicate its fundamental differences from the well known identity of Perl, major versions are now available again for Perl to leverage, and it is a very important step to show that the language is still developed and used. I completely agree with the motivation and ideals presented, and have thought a lot about the benefits and risks involved in such ideas long before I was aware of this project.

I do not generally work with ancient code that uses ancient practices. I work with CPAN modules that maintain compatibility with wider or narrower ranges of Perl versions for various reasons. I work with modern code for my own and business use that already will not function on older Perls. I work with newcomers that have written code based on modern Perl tutorials, and newcomers that have written code based on ancient Perl tutorials. It's from this perspective that I evaluate the proposed direction, the stated goal of which is to optimize for new users and active maintainers over abandoned code.

The announced direction

I am sure there are plenty of well studied mechanisms for evaluating proposals of such a nature, but I am a simple software engineer, so I will analyze the risks and benefits of the approach. This is an opinionated process, but I have tried to be fair and consider the honest potential of the currently stated plans.

Unequivocally, I will not consider ideas here that effectively result in forking Perl. These include ideas to intentionally be incompatible with CPAN, or use file extensions like .pm7 to indicate the incompatibility. We have the most useful possible evidence in where this leads thanks to Raku. This also does not take into account various milder assumptions people have made about implementation, because no such details have so far been announced.

The benefits of the announced direction:

• The use of major versions allows for a clear message to those outside the Perl bubble about continued development of Perl and easy to remember milestones for when features are introduced or removed.
• New defaults will signal to the user what style is considered modern.
• New code, written by either newcomers or experienced Perl coders, will not need to consider what set of esoteric pragmas to enable to get a modern style, as they will be enabled by default.
• Code can be written to take advantage of new features without lots of boilerplate or restricting the code from running on older versions; this benefit only really applies to strict and warnings, since any other features create a de facto minimum requirement of the Perl version they were introduced.

The risks of the announced direction:

• CPAN modules which largely support more than only the latest version of Perl will not be able to make use of Perl 7 changes.
• Perl users who miss the Perl 7 announcement or use old tutorials will continue to miss the existence of new features.
• Various tools and modules assuming a major version of 5 will need their assumptions fixed.
• Newcomers using tutorials written for Perl 5, Perl 5 books, code on Stack Overflow, etc may not understand why the code does not work on Perl 7 or how to fix it.
• Code written for Perl 7 will contain no indication of the assumptions it has made which will be broken (often silently, due to the nature of the proposed new defaults) if it is run in Perl 5; and will have no indication of the minimum version required to support the features it is using.
• The relationship between the ":7.0" feature bundle and the features enabled by default is currently not specified. If they are to be the same, important features for writing modern code that are problematic as defaults such as 'unicode_strings' and 'signatures' cannot ever be enabled in feature bundles, thus continuing the current problem of having to use esoteric boilerplate for modern code. If they are to be different, the implementation and presentation of this distinction will be necessarily very complex. (UPDATE: It is now planned for the feature bundle to be separate from the features enabled by default.)
• The impact on oneliners is yet to be explored. The -E option is commonly used to enable the most current feature bundle, but not strict, because while the feature bundle is quite unlikely to break oneliners in comparison to full scripts and modules, strict is generally only annoying in a oneliner context. Either -e or -E enabling strict could significantly affect Perl's use as a sysadmin and ops tool; and -e enabling unexpected features when being used specifically to avoid that would be another symptom of the proposed incompatibility.
• Large amount of CPAN modules will not work in Perl 7; plans for working around this would either involve every affected CPAN author, which is a virtual impossibility for the stated 1 year time frame; or the toolchain group, a loose group of people who each maintain various modules and systems that are necessary for CPAN to function, who either have not been consulted as of yet or have not revealed their plans related to the tools they maintain. Going into this potential problem sufficiently would be longer than this blog post, but suffice to say that a Perl where highly used CPAN modules don't seamlessly work is not Perl.
• (UPDATE) Perl code analysis tools will not be able to differentiate code intended to be run on Perl 7 or Perl 5 without an explicit code declaration to indicate the differing assumptions. Shebangs are only present in complete script files not modules or code snippets, and are an unreliable indicator (even if a new interpreter is needed, there is no requirement that a Perl 7 interpreter includes the 7 in its name). File extensions are once again only a property of full files rather than code snippets, and requiring that all Perl 7 code use separate file extensions would also mean that all existing Perl code does not work in Perl 7.
• Downstream vendors will need to evaluate how to package Perl going forward. Since Perl 7 would be incompatible, the most likely option is maintaining a separate Perl 7 binary and leaving Perl 5 in place for the foreseeable future, similar to the Python 2 to 3 transition. They will need to be aware of the strategy chosen to make CPAN modules work with Perl 7 and employ a solution of their own for the CPAN modules they package as well as the CPAN installers they provide.
• Active core developers, toolchain developers, CPAN authors, and downstream packagers are currently relatively small groups of people, thus the burden on each of these groups to solve the respective problems is that much greater, risking further burnout from these essential workers of the Perl ecosystem.
• Perl development will be frozen for the period of time until Perl 7 is released, which has currently been specified as from 6 months to a year; we are very familiar with the dangers of pausing development of Perl in anticipation of a new version. (UPDATE: Perl development is no longer frozen, though the initial Perl 7 release is still not planned to incorporate current development.)
• The culture and assumptions of the Perl stability policy will be fundamentally challenged through each aspect of the ecosystem. This may be the intent, but it is still a risk.
• Critics will point out that Perl 7 breaks code and does not contain anything new.

You may notice that I did not include "Perl 7 allows cruft to be removed, simplifying the language implementation" as a benefit, because this is already done in every yearly release and not changed in any way by the announced direction, except for the improved ability to advertise such changes by associating them with major version releases.

I also did not include the benefit of "New defaults encourage users to stop using old features and syntax in existing code", because in practice nothing encourages this except deprecating or removing the ability to use such syntax, which again does not significantly change under this direction.

A slightly different proposal

Let's examine the risks and benefits of a slightly different proposal, as follows.

Perl development continues in the normal fashion in the Perl 5.33 development track, and the next major release is 7.0.0 instead of 5.34.0. This version's feature bundle is enhanced to turn off the 'indirect' feature, and possibly to enable new features that have been added in this year of development; there are already rough plans for try/catch and multidimensional features, and the signatures feature could be moved towards completion. In addition, "use v7" or higher will enable the warnings pragma, and possibly the utf8 pragma for source encoding, in addition to strict that is already enabled for "use v5.12" or higher. Old features and syntax can be removed through the normal policies if deprecated, or the deprecation process can be started for others. Crucially, the default state of the interpreter does not set any features or pragmas differently from Perl 5, barring removal of deprecated syntax as in previous releases.

The benefits of this proposed direction:

• The use of major versions allows for a clear message to those outside the Perl bubble about continued development of Perl and easy to remember milestones for when features are introduced or removed. (same as announced)
• The "use v7" pseudo-pragma will signal to the user what style is considered modern.
• New code, written by either newcomers or experienced Perl coders, will not need to consider what set of esoteric pragmas to enable to get a modern style, as it will be represented by "use v7" in a way that "use VERSION" previously was inadequate for. (same as announced)
• Code written for Perl 7 will contain a clear indication of its assumptions and be restricted to running on that version or higher.
• The standard development process for the next release of Perl will have the added benefit of a major version for improved messaging of any significant features.

The risks of this proposed direction:

• CPAN modules which largely support more than only the latest version of Perl will not be able to make use of Perl 7 changes. (same as announced)
• Perl users who miss the Perl 7 announcement or use old tutorials will continue to miss the existence of new features. (same as announced)
• Various tools and modules assuming a major version of 5 will need their assumptions fixed. (same as announced)
• New code will need to include "use v7" to get a modern featureset.
• Critics will point out that Perl 7 does not take any chances.

Conclusions

It is easy to frame this discussion as being for or against progress, but please keep in mind I am not motivated to keep Perl 4 code running forever on new Perls, or to stand in the way of progress, but instead to focus efforts in order to improve the chances of successful progress. I completely support the consideration of incompatible changes (as, I will note again, is already part of the Perl development policy) provided I believe they will have a proportional potential for benefit to the ecosystem. I absolutely want Perl 7 to be a positive message for the continued development and usage of Perl, and to encourage changes to how we think about using the language as a whole.

While I recognize that there are potential benefits that only changing the default features can provide, I encourage the reader to consider the practical fallout of changing these assumptions in each component of the Perl ecosystem, and what it will look like if we were to consider doing this again for each major version following. The proposal suggests it would be the first of many such breakages, and even if we only do this once, it is not just a one time change to adapt to, but rather has lasting implications for the language and its users for the foreseeable future.

I can only conclude in this case, as I have for the past few years watching Raku grow into its own identity, that the significant risks and unknowns unique to the decision to incompatibly change feature defaults are a cost not worth the benefits for Perl. The announced direction does not solve many of the stated problems, and most of the same benefits can be achieved with nearly the standard development process, by leveraging the newly available major versions in messaging, and requiring "use v7" to activate the new normal.

Reddit comments

UPDATE 2020-08-13: Added risk regarding code analysis tools, and some updates to existing risks based on current discussions.

The official Perl documentation site at https://perldoc.perl.org was recently overhauled. Independently, I put together a reimagined documentation site that would be hosted at https://perldoc.pl. In the interest of providing a documentation site that best serves the needs of the Perl community, we need your feedback. Please give both sites a try, in-depth if you want, or just how you would most commonly use the site. What do you like about the design or the functionality of each site? What is missing or can be improved? Any feedback regarding what you want from the Perl documentation site is helpful and appreciated. Please leave comments here or in the linked posts by Monday Nov 18th.

Reddit comments

PerlMonks comments

CPAN is wonderful and it is vast. Task::Kensho offers a curated look at the best it has to offer for those who don't know what to look for. But to remain useful, it must keep up with the trends of CPAN and the community. Thus, the community's input is vital to its maintenance.

Please, take a moment and look through the open issues. Comment or add a reaction in support of changes that make sense to you, and open a new issue if you think something is missing.