Perl 6 Archives

The Missing Contributors of Perl 6

Read this article on Rakudo.Party

Today, I came across a reddit post from a couple months back, from a rather irate person claiming themselves to be possibly the only person to never receive any credit for their work on Perl 6.

I was aware that person committed at least one commit and knowing the contributors list is generated automagically with a script, I thought to myself "Well, that's clear and provable bullshit." And I went to prove it.

Moar No More

I looked up the commit I knew about, looked at the release announcement for the release it went into and… that person was indeed missing! It was the 2017.02 release, which I released. So what was going on? Did I have an alter-ego that shamelessly erased random people from the contrib list without my having any memory of it?!

First, a brief intro on how the contrib script works: it uses git to look up commits in checkouts of 5 repos: Rakudo, NQP, MoarVM, Docs, and Roast. Until December 2016 the script just used the day of the release as last release, which was later switched to using the timestamp on the Rakudo's tag. The script gathers all the contributors from commits, crunches the names through the names map in CREDITS files in the repos, and it spits out the names ordered by the number of commits made, largest first.

I set out to figure out why a person was missing from the release announcement. After digging through commits, CREDITS files, and tracing the code in the contributor generating script, I found out that in September 2016, I introduced a bug into the contributors script. After some refactoring I accidentally left out MoarVM repository from the list of repos the script searches, so all the contributors to the MoarVM since September 2016 were missing! Since many of them also contribute to the other 4 repos, it was harder to spot that something was wrong.

I filed the problem as R#2024 and left it at that for the time being.

Missing More

I started working on the problem and implemented a new feature in the contributors script that lets you look up contributors for past releases. Neat! So let's try it out for some release before my bug was made, shall we?

I ran the contrib script for 2016.08 relase and then ran another script that diffed the names from that output against what is on the release announcement. The output was:

Announcement has these extra names: David Warring
Contrib script has these extra names: Arne Skjærholt, Bart Wiegmans

The announcement had an extra name and was missing two. The way the contrib script figures out when one release ends and another starts is iffy, especially so in the past. There's a gap of about a day where contributors can slip through: e.g. release manager runs the release at 2PM, someone commits at 3PM, and that commit didn't make it into this release and will be included in the next, or might even be missed entirely.

So that was one problem I noticed. Is that where the difference for 2016.08 release names list comes from? Let's try the earliest post-The-Christmas release: 2016.01-RC1

Announcement has these extra names: Andy Weidenbaum, Lloyd Fournier, skids
Contrib script has these extra names: A. Sinan Unur, Aleks-Daniel Jakimenko-Aleksejev,
Brad Gilbert, Brian S. Julin, Brock Wilcox, Bruce Gray, Carl Masak,
Christian Bartolomäus, Christopher Bottoms, Claudio Ramirez, Dale Evans,
Dave Olszewski, David Brunton, Fritz Zaucker, Jake Russo, James ( Jeremy )
Carman, Jeffrey Goff, Jim Davis, John Gabriele, LLFourn, Marcel Timmerman,
Martin Dørum Nygaard, Neil Shadrach, Salvador Ortiz, Shlomi Fish, Siavash
Askari Nasr, Stéphane Payrard, Sylvain Colinet, Wenzel P. P. Peppmeyer,
Zoffix Znet, fireartist, sylvarant, vinc17fr

That's huge! One name stood out to me in that list—and it isn't my own—it was that same person from reddit who was complaining that they don't get credit. They got left out twice: in 2016.01 and again in 2017.02. No wonder they're pissed off, but I wish they would've said something in 2016.01, so we'd've fixed the Missing Persons issues back then instead of now.

The 2016.02 release has a bunch of missing names as well. I can surmise the cause of the issue is a previously fixed mis-implementation of the contributors script where it'd be quiet if some of the repo checkouts were missing. Neither I (until that point), nor earlier release managers had all of at the right locations the script was expecting, so it's possible that's how some repos were missed.

At the time, I assumed only the docs repo was missing and we credited missing Docs contributors in the 2016.08 announcement. However, now I realize that other release managers likely had different directory setups and thus missed different sets of people.

The Future

Thus, we have identified four issues with the way contributor's script is or has been generating the list of contributors:

  • Relying on the time when the release manager runs the contributor script, potentially creating a gap of unrecorded contributions between the time the script is run and the time the next run of contrib script considers as "last release"
  • Relying on release manager's setup of directories/repos. Even after a previous fix in this area, we're still relying on the release manager to have up-to-date checkouts of repos
  • Missing contributors from entire repositories due to unnoticed bug in the code
  • What happens with commits made at the time of past release in a branch that is merged at the time of the next release? Do they get lost?

I'm taking the lazy way out and leaving it to the current release managers to resolve these problems. I filed R#2028 with the list of issues and have full trust the solution that will be implemented will be suitable :)

The Found

And now, of course, the list of previously unsung heros who made Perl 6 better in the past two and a half years, in alphabetical order. I've also added them to our past release announcements.

It's possible this list includes the missing-found from 2016.08 announcement as well as people who were not logged in the CREDITS file in the past but are now, but I figure it's better to list them twice than none at all.

If you still believe we're missing someone, let us know so the problem can be fixed.

2016.01/2016.01-RC1

A. Sinan Unur, Aleks-Daniel Jakimenko-Aleksejev, Brad Gilbert, Brian S. Julin, Brock Wilcox, Bruce Gray, Carl Masak, Christian Bartolomäus, Christopher Bottoms, Claudio Ramirez, Dale Evans, Daniel Perrett, Dave Olszewski, David Brunton, Fritz Zaucker, Jake Russo, James ( Jeremy ) Carman, Jeffrey Goff, Jim Davis, John Gabriele, LLFourn, Marcel Timmerman, Martin Dørum Nygaard, Neil Shadrach, Salvador Ortiz, Shlomi Fish, Siavash Askari Nasr, Stéphane Payrard, Sylvain Colinet, Wenzel P. P. Peppmeyer, Zoffix Znet, fireartist, raiph, sylvarant, vinc17fr

2016.02

Bart Wiegmans, Brian S. Julin, Brock Wilcox, Daniel Perrett, David Brunton, Eric de Hont, Fritz Zaucker, Marcel Timmerman, Nat, Pepe Schwarz, Robert Newbould, Shlomi Fish, Simon Ruderich, Steve Mynott, Wenzel P. P. Peppmeyer, gotoexit, raiph, sylvarant

2016.03

Ahmad M. Zawawi, Aleks-Daniel Jakimenko-Aleksejev, Bahtiar kalkin- Gadimov, Bart Wiegmans, Brian S. Julin, Brock Wilcox, Claudio Ramirez, Emeric54, Eric de Hont, Jake Russo, John Gabriele, LLFourn, Mathieu Gagnon, Paul Cochrane, Siavash Askari Nasr, Zoffix Znet, jjatria, okaoka, sylvarant

2016.04

Brian S. Julin, Brock Wilcox, Christopher Bottoms, David H. Adler, Donald Hunter, Emeric54, Itsuki Toyota, Jan-Olof Hendig, John Gabriele, Mathieu Gagnon, Nick Logan, Simon Ruderich, Tom Browder, Wenzel P. P. Peppmeyer, Zoffix Znet

2016.05

Aleks-Daniel Jakimenko-Aleksejev, Brian Duggan, Brian S. Julin, Brock Wilcox, Christopher Bottoms, Clifton Wood, Coleoid, Dabrien 'Dabe' Murphy, Itsuki Toyota, Jan-Olof Hendig, Jason Cole, John Gabriele, Mathieu Gagnon, Philippe Bruhat (BooK), Siavash Askari Nasr, Sterling Hanenkamp, Steve Mynott, Tadeusz “tadzik” Sośnierz, VZ, Wenzel P. P. Peppmeyer, Will Coleda

2016.06

(contrib script missing repos issue is fixed around this point, so the number of missing persons drops. Remaining ones are likely the ones that fell into the gap between releases; particularly MoarVM and docs contributors)

Itsuki Toyota, Matthew Wilson, Will Coleda, parabolize

2016.07

Bart Wiegmans, Brian S. Julin, Daniel Perrett, David Warring, Dominique Dumont, Itsuki Toyota, thundergnat

2016.08

Arne Skjærholt, Bart Wiegmans

2016.09

(missing MoarVM bug is introed at this point; we start to see the missing MoarVM devs who mostly work on MoarVM and not other repos. Also a bunch of docs people who likely fell into the gap between releases)

Alexey Melezhik, Bart Wiegmans, Paul Cochrane

2016.10

Brent Laabs, Jimmy Zhuo, Steve Mynott

2016.11

Bart Wiegmans, Itsuki Toyota, Jimmy Zhuo, Mark Rushing

2016.12

Bart Wiegmans, Jimmy Zhuo, LemonBoy, Nic Q, Reini Urban, Tobias Leich, ab5tract

2017.01

Antonio Quinonez, Jimmy Zhuo, M. Faiz Zakwan Zamzuri

2017.02

A. Sinan Unur, Bart Wiegmans, Benny Siegert, Jeff Linahan, Jimmy Zhuo, Lucas Buchala, M. Faiz Zakwan Zamzuri

2017.03

Jonathan Scott Duff, Lucas Buchala, Moritz Lenz

2017.04

Bart Wiegmans, eater

2017.05

Bart Wiegmans, Paweł Murias

2017.06

Bart Wiegmans, Jimmy Zhuo, Oleksii Varianyk, Paweł Murias, Robert Lemmen, gerd

2017.07

Bart Wiegmans, Douglas Schrag, Gerd Pokorra, Lucas Buchala, Paweł Murias, gerd

2017.08

Bart Wiegmans, Dagfinn Ilmari Mannsåker, Douglas L. Schrag, Jimmy Zhuo, Mario, Mark Montague, Nadim Khemir, Paul Smith, Paweł Murias, Philippe Bruhat (BooK), Ronald Schmidt, Steve Mynott, Sylvain Colinet, rafaelschipiura, ven

2017.09

Bart Wiegmans, Dan Zwell, Itsuki Toyota, Jan-Olof Hendig, Jimmy Zhuo, Mario, Paweł Murias, Rafael Schipiura, Skarsnik, Will Coleda, smls

2017.10

Bart Wiegmans, Jimmy Zhuo, Joel, Julien Simonet, Justin DeVuyst, M, Mario, Martin Ryan, Moritz Lenz, Patrick Sebastian Zimmermann, Paweł Murias, bitrauser, coypoop, eater, mryan, smls

2017.11

Bart Wiegmans, Jimmy Zhuo, Martin Barth, Patrick Zimmermann, Paweł Murias

2017.12

Bart Wiegmans, Paweł Murias, Stefan Seifert, brian d foy

2018.01

Bart Wiegmans, Daniel Dehennin, Paweł Murias, Stefan Seifert, Will Coleda

2018.02

Bart Wiegmans, Daniel Green, Paweł Murias, cygx, wukgdu

2018.03

Bart Wiegmans

2018.04

Bart Wiegmans, Paweł Murias, cc, gerd

2018.05

Antonio, Bart Wiegmans, elenamerelo

2018.06

Bart Wiegmans, JJ Merelo

Conclusion

So this was quite a fun investigation and hopefully all the missing people have been found and this is the last missing-found persons list we compile.

The most important lesson, however, is: report problems as soon as you find them. We could've fixed this at the start of 2016, and those who knew they were left out could've saved two years of being upset about it.

-Ofun

WANTED: Perl 6 Historical Items

Read this article on Rakudo.Party

The Perl 6 programming language had a turbulent birth. It was announced in the summer of 2000 and the first stable language release shipped out only 2 years ago, on Christmas, 2015. A lot has happened during that decade and a half, yet the details are hard to piece together.

After my recent facelift to rakudo.org, I'm working on a (second) facelift to perl6.org website.

Part of the work involves bringing all the Perl 6 deliverables under one umbrella, so the user isn't thrown around multiple websites, trying to find what to install. At the same time, we want to strengthen the distinction between Perl 6 the language and the compilers that implement it, as well as encourage more implementors to give it a go at implementing a Perl 6 programming language compiler.

The Perl 6 Programming Language Museum will be part of that effort and along with interesting tidbits of Perl 6 history, it'll showcase past implementation attempts that may no longer be in active development today. Since I don't know much about what happened before I came to the language sometime in 2015, I need your help in collecting those tidbits.

Larry Wall at FOSDEM 2015, photo by Klapi

In my mind's eye, I'm imagining a few pages on perl6.org; something in the same vein as Computer History Museum's pages—pictures, years, and info, and potentially links to code repositories. Depending on the content we collect, it's possible there will be a digital PDF version of the Museum that can also be printed and handed out at events, if desired.

I'm looking for:

  • Descriptions of interesting/significant events (like the mug throwing incident).
  • Descriptions of interesting/significant implementations of Perl 6 or influential Perl 6 projects. Having links to repos/tarballs of their code is a plus.
  • Samples of interesting/significant email threads or chat logs.
  • Pictures of interesting/significant objects (first sight at plush Camelias?).
  • Pictures of interesting/significant humans (a filled out model release form is required).
  • Anything else that's Museum worthy.

If you have any of these items, please submit them to the appropriate year directory in the Perl 6 Museum Items repository. If you're a member of Perl 6 GitHub org, you should already have a commit bit to that repo. Otherwise, submit your items via a pull request.

Let's build something cool and interesting for the people using Perl 6 a hundred years from now to look at and remember!

If you have any questions or need help, talk to a human on our IRC chat.

-OFun

Perl 6: On Specs, Versioning, Changes, and... Breakage

Read this article on Rakudo.Party

Recently, I came across a somewhat-frantic comment on StackOverflow that describes a 2017.01 change to the type of return value of .sort:

"you just can't be sure what ~~ returns" Ouch. […] .list the result of a sort is presumably an appropriate work around. But, still, ouch. I don't know of a blog post or whatever that explains how P6 approaches changes to the language; and to roast; and to Rakudo. Perhaps someone will write one that also explains how this aspect of 2017.01 was conceived, considered and applied; what was right about the change; what was wrong; etc.

Today, I decided to answer that call to write a blog post and reply to all of the questions posed in the comment, as well as explain how it's possible that such an "ouch" change made it in.

On Versioning

The '6' in Perl 6 is just part of the name. The language version itself is encoded by a sequential letter, which is also the starting letter of a codename for that release. For example, the current stable language version is 6.c "Christmas". The next language release will be 6.d with one of the proposed codenames being "Diwali". The version after that will be 6.e, then 6.f, and so on.

If you've used Perl 6 sometime between 2015 and 2018, you likely used the "Rakudo" compiler, which is often packaged as "Rakudo Star" distribution and is versioned with the year and the month of the release, e.g. release 2017.01.

In some languages, like Perl 6's sister language Perl 5, what the compiler does is what the language itself is. Bugs aside, if the latest (2017.09) Perl 5 compiler gives 4 for 2+2, then that's the definition of what 2+2 is in the Perl 5 language.

In Perl 6, however, how a compiler (e.g. "Rakudo") behaves or what it implements does not define the Perl 6 language. The Perl 6 language specification does. The specification consists of a test suite of about 155,000 tests and anything that passes that test suite can call itself a "Perl 6 compiler".

It's to this specification version 6.c "Christmas" refers. It was released on December 25, 2015 and at the time of this writing, it's the first and only release of a stable language spec. Aside from a few error corrections, there were no changes to that specification… The latest version of Rakudo still passes every single test—it's a release requirement.

On Changes

Ardent Perl 6 users would likely recall that there have been many changes in the Rakudo compiler since Christmas 2015. Including the "ouch" change referenced by that StackOverflow comment. If the specification did not change and core devs are not allowed to make changes that break 6.c specification, how is it possible that the return type of .sort could have changed?

The reason is—and I hope the other core devs will forgive me for my choice of imagery—the specification is full of holes!

It doesn't (yet) cover every imaginable use and combination of features. What happens when you try to print a Junction of strings? As far as 6.c version of Perl 6 language is concerned, that's undefined behaviour. What object do you get if you call .Numeric on an Rat type object rather than an instance? Undefined behaviour. What about the return value of .sort? You'll get sorted values in an Iterable type, but whether that type is a Seq or a List is not specified by the 6.c specification.

This is how 2017.01 version of Rakudo managed to change the return type of .sort, despite being a compliant implementation of the 6.c language—the spec was not precise about what Iterable type .sort must return; both Seq and List are Iterable, thus both conform to the spec. (It's worth noting that since 2017.01 we implemented an extended testing framework that also guides our decisions on whether we actually allow changes that don't violate the spec).

In my personal opinion, the 6.c spec is overly sparse in places, which is why we saw a number of large changes in 2016 and early 2017, including the "ouch" change the commenter on StackOverlow referred to. But… it won't stay that way forever.

The Future of the Spec

At the time of this writing, there have been 3,129 commits to the spec, since 6.c language release. These are the proposals for the 6.d language specification. While some of these commits address new features, a lot of them close those holes the 6.c spec contains. The main goal is not to write a "whole new spec" but to refine and clarify the previous version.

Thus, when 6.d is released, it'll look something like this:

A few more slices of new features, but largely the same thing. Still some holes (undefined behaviour) in it, but a lot less than in 6.c language. It now defines that printing a Junction will thread it; that calling .Numeric on a Numeric type object gives a numeric equivalent of zero of that type and a warning; and that the .sort's Iterable return type is a Seq, not a List.

As more uses of combinations original designers haven't thought of come around, even more holes will be covered in future language versions.

Breaking Things

The cheese metaphor covers refinements to the specification, but there's another set of changes the core developers sometimes have to make: changes that violate previous versions of the specification. For 6.d language, the list of such changes is available in our 6.d-prep repository (some of the listed changes don't violate 6.c spec, but still have significant impact so we pushed them to the next language version).

This may seem to be a contradiction: didn't I say earlier that passing 6.c specification is part of the compiler's release requirements? The key to resolving that contradiction lies in ability to request different language versions in different comp units (e.g. in different modules) that are used by the same program.

A single compiler can support multiple language versions. Specifying use v6.c pragma loads 6.c language. Specifying use v6.d (currently available as use v6.d.PREVIEW) loads 6.d language. Not specifying anything loads the newest version the compiler supports.

One of the changes between 6.c and 6.d languages is that await no longer blocks the thread in 6.d. We can observe this change using a single small script that loads two modules. The code between the two modules is the same, except they request different language versions:

# file ./C.pm6
use v6.c;
sub await-c is export {
    await ^10 .map: {
        start await ^5 .map: { start await Promise.in: 1 }
    }
    say "6.c version took $(now - ENTER now) secs";
}

# file ./D.pm6
use v6.d.PREVIEW;
sub await-d is export {
    await ^10 .map: {
        start await ^5 .map: { start await Promise.in: 1 }
    }
    say "6.d version took $(now - ENTER now) secs";
}

# $ perl6 -I. -MC -MD -e 'await-c; await-d'
# 6.c version took 2.05268528 secs
# 6.d version took 1.038609 secs

When we run the program, we see that no-longer blocked threads let 6.d version complete a lot faster (in fact, if you bump the loop numbers by a factor, 6.d would still complete, while 6.c would deadlock).

So this is the Perl 6 mechanism that lets the core developers make breaking changes without breaking user's programs. There are some limitations to it (e.g. methods on classes)—so for some things there still will be standard deprecation procedures. We also try to limit the number of such spec-breaking changes, to reduce the maintenance burden and impact on users who don't want to lock their code down to some older version. Thus, don't worry about getting some weird new language on the next language release—the differences will be minimal.

Who Decides?

This all brings us to one of the questions posed by that StackOverflow user: how do language changes get conceived, considered, and applied—in short: who decides what the behaviour is to be like? What is the process?

As far as conception goes, many of the current ideas are based on seeing what our users need. Some proposals come directly from users; others get inspired as more elegant solutions to problems users showed they were trying to solve. Some of the changes proposed for 6.d language were informed by problematic areas of currently-implemented features that weren't foreseen during original implementation.

When it comes to implementation, the scope of the feature and core developer's expertise with the given area of the codebase generally drive the process. With the "ouch" change, the expert in the area of Iterables deemed Seq to be a superior type for .sort to return, due to its non-caching behaviour as well as its ease of degenerating into a caching List.

Some changes get opened as an Issue on the bugtracker first, to notify other devs of the impending change. Large changes usually get a proposed design written down first. The proposal is shared with the core devs and feedback is gathered before the proposal is actually implemented. The implementation of significant things is also merged far away from the date of the next release, to let the bleeding-edge users find any potential problems in the work.

Geth, our IRC bot, announces all commits in our development IRC channel. Most of the core devs backlog that channel, so any of the potentially problematic commits—even if one of the devs goes ahead and commits the change—get discussed and at times reverted.

The Perl 6 pumpking (Jonathan Worthington) and the BDFL (Larry Wall) are available to provide feedback on controversial, questionable, or large changes being proposed. They also have the veto power on any changes. Our messaging bot helps us request feedback from them, even if they're currently not in the chat.

When it comes to errata to previous specifications, unless the test to be changed is "obviously wrong", the decision on whether the errata can be applied is delegated to the Release Manager (AlexDaniel), and informed by the pumpking/BDFL, if required.

The Future

The current process is a bit loose in places. A test that's "obviously wrong" to one person might have some valid reasons behind it to someone else. This is why the TODO for 6.d release lists several documents to be written that will refine the procedures for various types of changes.

It won't be on the scale of PEP, but simply something more concrete for the core devs to refer to, when performing changes that have some impact on the users. It's a balancing act between organization and procedure and letting through a consistent flow of contributions.

And if breaking changes have to be made, an alert will be pushed to the the P6lert service for users of Perl 6 to get informed of them in advance.

Conclusion

Today, we gleaned an insight into how Perl 6 core devs introduce changes to the compiler and the language.

The language specification and the compiler's behaviour are separate entities. The 6.c language specification has places of unspecified behaviour, which is how changes that have large impact on the users slipped through in the past.

The extended testing framework as well as specification clarifications offered by 6.d language proposal tests that refine the specification and close the holes with undefined behaviour reduce unforeseen impact on the users.

The core dev team informs their decisions based on user's feedback and the way the language is used by the community. Large changes get written up as proposals and the pumking/BDFL offer advise on anything controversial.

In the future, more refined practices for how changes are made will be defined, as we work on making upgrade experience more predictable and non-breaking for our users. The P6lert service helps that goal and is already available today.

Hope this answers all the questions :)

Announcing P6lert: Perl 6 Alerts Directly From Core Developers

Read this article on Rakudo.Party

Development of Rakudo Perl 6 is quite fast-paced, with hundreds of commits made each month to its five core repositories. Users undoubtedly feel some impact from those commits: bug fixes may break code that relied on them, backend changes may have unforeseen impact on the user code, new useful features may be implemented that users would want to know about.

In the past, for things with very large impact, we made blog posts, but there are lots of small things that fly under the radar, unless you actively pay a lot of attention to Rakudo Perl 6's core development.

To help all of our users to be aware of important issues, we're announcing introduction of P6lert service: tweet-sized alerts from Perl 6 Core Developers.

The Goods

The P6lert service primarily consists of alerts.perl6.org website, but with it come a variety of ways to receive alerts posted on it:

The Content

While we'll make adjustments as we move forward, I foresee most of the non-critical alerts will largely include things that are: (a) more important than simply hoping users-who-care will read about it in the ChangeLog; (b) not as important to warrant a notification blog post.

As a rule-of-thumb, if you picture a user who added p6lert script to their compiler upgrade procedure, the alerts the script will show will inform that user on everything they need to know to perform that upgrade safely.

The alerts are also deliberately length-limited to be easy to process and fast to digest. As they're posted via an IRC bot, the poster has at most about 400 characters to work with.

Each alert has an affects field for it, giving additional info what the alert applies to. I think it'll often be empty, as alerts affecting latest compiler versions imply they affect whatever latest release is at the time alert was posted.

The alerts have a severity rating: low, normal, and high indicating how important an alert is. Along with those, come two out-of-band ratings: info and critical. Info alerts will usually be something the users don't need to act upon, while critical alerts will often simply contain a link to a blog post that details a critical issue.

Of the top of my head, here are some real-life examples from the past and how I'd rate their severity on the P6lert service:

  • info - Telemetry implementation. This was a fairly large implementation of a feature in the core and some users may be interested in it. At the same time, they don't have to act on this alert. Rakudo and Rakudo Star release may also be info alert material.
  • low - implementation of output buffering on IO handles. Once that was implemented, we noticed some minor fallout in code that assumed lack of buffering. A low-severity alert could notify users about this.
  • normal - A real-life normal-severity alert is already posted on the site. During 6.d spec pre-release review, the Str.parse-names method was placed under deprecation and its functionality was moved to live under Str.uniparse. While this method always existed as a 6.d-proposal, it's known to us that some users already use it and an alert will help them ensure their code keeps working past 6.e language release.
  • high - a while ago, .subst was briefly made to die if it couldn't write to $/; same as some methods already do. Upon examination of ecosystem fallout, this change was reverted, pending further review. However, were it to stay, a high-severity alert would be in order.
  • critical - when we finished work on lexical require, conditional loading of modules many users used was silently failing and the users needed to change their code to correct reliance on the old, buggy behavour. We issued a blog post with upgrade instructions. A critical alert would be a link to this blog post.

That's my vision for host the system will be used, but it'll evolve to suit our needs as more core devs and more of our users start using it. The core dev docs for the system along with code for all the pieces is available in perl6/alerts repo.

Conclusion

As part of improving using user experience, Perl 6 core devs now offer alerts.perl6.org service that will list important information about latest developments in the land of Perl 6. There are numerous ways to consume those alerts, such as an RSS or Tweeter feeds, a command line utility, and an easy-to-use API.

The alerts will come in 5 different severity ratings, indicating their importance. We'll continue to improve the system to best suit our users' needs.

If you have any questions or feedback, you can always talk to the core devs on #perl6-dev IRC chat.

-Ofun

CPAN6 Is Here

Read this article on Rakudo.Party

If you've been following Rakudo's development since first language release on Christmas, 2015, you might've heard of numerous people working to bring CPAN support to Rakudo Perl 6.

Good news! It's finally here in usable form and you should start using it!

Let's talk about all the moving parts and how to upload your dists to CPAN.

The Moving Parts and Status Report

All of the heavy lifting has been done awhile back, during Perl Toolchain Summit and other times. I wasn't present for it to know the details, but to catch up you could join #perl6-toolchain chat and talk to humans or read the channel log. PAUSE/CPAN support for Perl 6 dists was implemented and zef module installer was trained to check for CPAN dists as well as our GitHub/GitLab-based ecosystem (called "p6c").

The only bit that was left missing is a front-end to browse available CPAN dists. There is a team who wished to take metacpan.org's codebase and modify it for Rakudo dists. I'm told that project is currently "stalled but not dead".

That's unfortunate, however, earlier this week, modules.perl6.org was taught to handle CPAN dists, so—hooray!—we finally have some sort of a front-end for CPAN dists. If you only want to see CPAN dists in search results, you can use from:cpan search qualifier (just like you can use from:github and from:gitlab ones).

GitHub/GitLab dists URLs still direct to repos, but CPAN dists have a file browser that lets you see what files are up in the dist. The file browser also renders README.md/README.markdown markdown readme files.

The viewer doesn't have all the bells and whistles of metacpan.org and doesn't (yet) render POD6, but it's certainly useable. The person who implemented this viewer will be busy preparing 6.d language release in the near future and won't have the time to make additional improvements to the CPAN dist viewer. So… you're invited to contribute and make it better!

Why Upload to CPAN

CPAN has many mirrors ensuring module installation is not affected whenever GitHub (a single website) has issues. The uploaded dists are also immutable and stay there forever (barring special deletion requests, even deleted dists remain available on BackPAN). This means people are more likely to trust these dists for use in their larger projects that need dependable dependencies. Lastly… it's what the cool kids use!

How to Upload to CPAN

Here's the process for how you can get your dists to CPAN. If these dists are currently listed in our p6c ecosystem, both p6c and CPAN versions will appear on modules.perl6.org, and you're encouraged to remove the p6c version. Some of the described tools are brand-new and others are brand-old, created before Rakudo existed, so treat this guide as part information and part invitation to improve the tools.

Step 1: Get a PAUSE account

PAUSE stands for "The [Perl programming] Authors Upload Server", it's located at pause.perl.org, and it's a site you use to upload dists to CPAN.

Go to request PAUSE account page and subscribe for an account. The "desired ID" field is for your PAUSE ID, and it's currently used as "author" field on modules.perl6.org. For example, mine is ZOFFIX.

I had my account for over a decade, so my memory is a bit fuzzy, but I think you'll need to wait for a human to approve and create your account—it's not instantaneous.

Step 2: Make a Dist Archive

You can manually create a tarball or a zip archive. I don't have all the details on which files you're supposed to have in them; you can take a look at other CPAN dists to see what they're doing or…

Use App::Mi6 module! It's possible you were already using it to create dists, in which case you're in luck, as you can just run mi6 dist to make a dist archive.

I rolled my dists by hand and wrote all the docs in README.md, so when I gave mi6 dist a whirl, it replaced my README.md with emptiness because I wasn't using any POD6—something (currently) to watch out for.

Step 3: Upload Your Dist

The first option is to upload manually: log into pause.perl.org, then go to Upload a file to CPAN, be sure to select Perl6 in the select input and then upload either via an uploaded file or a URL.

The second option is to use App::Mi6's mi6 upload command.

Shortly after the upload, you'll get an email about whether your upload succeeded (you can also see emails on nntp.perl.org). Make sure you have a META6.json file in your dist and that the dist version you're uploading is higher than the currently uploaded version. Those are the most common upload errors.

Step 4: Relax and Wait

If you're on IRC, in about 10 minutes after your upload, our buggable robot will announce it:

<buggable> New CPAN upload: Number-Denominate-1.001001.tar.gz by ZOFFIX
    https://www.cpan.org/authors/id/Z/ZO/ZOFFIX/Perl6/Number-Denominate-1.001001.tar.gz

In about 2 hours, the dist will also appear on modules.perl6.org. Its updater is started in a cron job on 20th and 40th minute of the hour (unless a job is already running) and it takes about 2 hours to finish each run.

Step 5: Celebrate with the Appropriate Amount of Fun

That's about it to the process. I foresee more tools will be created in the future to make the process even easier than it is today. If you have any questions or issues, just talk to a human or a robot on our #perl6 IRC channel.

Conclusion

CPAN support for Rakudo Perl 6 dists is now usably here. You're encouraged to upload your dists to CPAN, to grow a more dependable ecosystem. You're also invited to improve and create tooling that manages and displays CPAN uploads.

-Ofun

6lang: The Naming Discussion Update

Read this article on 6lang.Party

When a couple months ago I rekindled the naming debate—the discussion on whether "Perl 6" should be renamed—I didn't expect anything more than a collective groan. That wasn't the case and today, I figured, I'd post a progress report and list the salient happenings, all the way to my currently being the proud owner of 6lang.party domain name.

The "Rakudo" Language

The "new" name I mentioned in my original post was Rakudo. As many quickly pointed out, it wasn't the greatest of names because it was the name of an implementation. Yes, I agree, but originally I thought few, if any, would be on board with a new name, or extended name, and Rakudo was basically the only name people already were using, so it stood out as something that could be "hijacked."

The Blog Post Fallout

There was quite a bit of discussion on r/perl, r/perl6, and blogs.perl.org. The general mood among the Perl community members who aren't avid 6lang users was that the entirely new name was a good idea. However, the 6lang users, and especially core devs, overall, argued "Perl 6" still had some recognition benefits and should not be removed entirely.

The middle ground was aimed at then: extend the language name. The "official" name would be among the lines of "Blah Perl 6" and users opposed to the 4-letter swear word would just use the name extension on its own, while those who feel the original name has benefits can still reap them.

The decision on the naming extension was placed on the 6.d language release agenda, with the final call on whether and with what the name should to be extended to be done by Larry, when we cut the 6.d language release.

The 6lang

Fast-forward two months. A kind soul (thank you, by the way!) asked Larry what he thought about the naming debate during the last Perl Conference:

Larry opined that we could have other terms by which Perl versions or Perl distributions are marketed as. So that gives us an option to pick an alternative name to be the second name with any "official" standing. Personally, I really like this idea; even more than name extension, because should there indeed be more benefit to the name without "Perl" in it, the alternative name will naturally become the most-used one.

Another core dev, AlexDaniel++, coined an alternative name: spelt 6lang; can be pronounced as slang, if you want to be fancy. I really liked the name, so I jumped in and registered 6lang.party

<AlexDaniel> Zoffix++ for making me recognize the need for
     alternative name. For a long time I was against
<AlexDaniel> and honestly, I can start using something like 6lang
     right away. “Rakudo Perl 6” is infringing on
     language/compiler distinction so I'm feeling reluctant
<Zoffix> OK, I'll too start using 6lang
* Zoffix is now a proud owner of 6lang.party :D
<timotimo> wow
<AlexDaniel> that was quick

And a couple of hours later, our Marketing Department churned out a new poster:

The drawback is that the name can't be used as an identifier… and Larry doesn't think it's a terribly sexy name.

* TimToady notes that 6lang isn't gonna work anywhere an identifier
     needs a leading alpha
<TimToady> it's also not a terribly sexy name
<TimToady> I could go for something more like psix, "where the p is silent
     if you want it to be" :)

Although, on the plus side, the name has the benefit that alphabetically it sorts earlier than pretty much any other language.

<AlexDaniel> If we see “6lang” as a more marketable alternative, then
     the fact that some things may not parse it as an identifier
     practically does not matter. However, this little bit is quite useful:
<AlexDaniel> m: <perl5 golang c# 6lang ruby>.sort.say
<camelia> rakudo-moar 39a4b7: OUTPUT: «(6lang c# golang perl5 ruby)␤»
<AlexDaniel> :)
<AlexDaniel> .oO( AAAlang – batteries included )

To 6.d Release And Beyond

So that's where things progressed to so far. No official decisions have been made yet, but we're thinking about it and playing with the idea. The decision on the naming debate is to be made during 6.d release.

Having learned a painful lesson from The Christmas release, we're reluctant to put down any dates for 6.d release, but I suspect it'll be somewhere between the upcoming New Year's and It's-Ready-When-It's-Ready.

See you then \o

The Rakudo Book Project

Read this article on Rakudo.Party

When I first joined the Rakudo project, we used to say "there are none right now; check back in a year" whenever someone asked for a book about the language. Today, there's a whole website for picking out a book, and the number of available books seems to multiply every time I look at it.

Still, I feel something is amiss, when I talk to folks on our support chat, when I read blog posts about the language, or when I look at our official language documentation. And it's due to that feeling that I wish to join the Rakudo book-writing club and write a few of my own. I dub it: The Rakudo Book Project.


The Books

The Rakudo Book Project involves 3 main books—The White Book, The Gray Book, and The Black Book—as well as 2 half-books—The Green Book and The Cracked Book.

The White Book will aim to provide introductory material to the Rakudo language. The target audience will benefit from prior programming experience, but it won't be strictly necessary for computer-savy people. The target audience is "adept beginners", as some might call it.

The book will cover most of Rakudo's features a typical Rakudo programmer might use in their projects, but it won't cover every little thing about each of them. By the end of the book, the readers will have written several programming projects and will be comfortable making useful, real-world Rakudo programs. More in-depth coverage of the language will be provided by The Gray Book, which is what The White Book's readers would read next. The Black Book will reach even deeper, exploring all of the arcane constructs. The progression through the books can be thought of as a plant growing in a flower pot. Initially, the roots extend through a large area of the pot, but they don't go all the way to all the walls and are rather sparse. As the plant grows, more and more roots shoot out, covering more and more volume of the pot. Same is with the books; while reading The White Book alone will let the plant survive, the root coverage will be sparse. However, by the end of The Black Book, the reader will be an expert Rakudo programmer.

Those three books are the core of my planned project. They're supplemented by two half-books on each end of the knowledge spectrum. The Green Book will target absolute programming beginners and get them up to speed just enough so they would be able to comfortably continue their learning using The White Book. On the other end of the spectrum is The Cracked Book. It's a half-book that follows The Black Book and won't provide more advanced techniques per say, but rather arcane "hacks" or even "bad ideas" that one might not wish to use in real-life code but which nevertheless provide some insight into the language.

The Cracked Book is yet a faint glimmer of an idea. Whether it will actually be made will depend on how much more I will want to say after The Black Book is complete. The Green Book is currently a bit amorphous as well. I have a 12-year old sibling interested in computers, so The Green Book might end up being a Rakudo For Kids.

The likely order in which the books will be produced is White, Gray, Green, Black, and Cracked. It's an ambitious plan, and so I won't be making any promises for producing more than one book at a time. Thus, the current aim is to produce just The White Book.

The Price

The digital versions of the books will be available for free.

Since Rakudo development can always use more funding, I plan to run crowd-funding campaigns during each of the book's development. 100% of all the collected funds will be used to sponsor Rakudo work (sponsoring someone other than me, of course). The campaigns will start once half of the target book has been created and the backers will get early preview digital copies as the book is developed further, as well as honourable mentions as Rakudo sponsors in the book itself.

Thus, the first Rakudo Core Fundraiser will launch once I have the first half of The White Book finished. I'm hoping that will happen soon.

The Why

Other than the obvious reason why people write the books—giving an alternate take on the material—I'd like to do this to cross off an item off my bucket list. Having written a terrible non-fiction book, lackluster fiction book, and a decent illustrated children's book, I hope to add a great technical book to the list, to complete it. I figure, with 5 books to attempt it, I'll be successful.

As for my alternate take, I hope to squash the myth that Rakudo is too big to learn as well as carve out a well-defined path for learners to follow. Just as I could make a living 10 years ago, when I barely spoke English, so a beginner Rakudo programmer can make useful programs with rudimentary knowledge of the language. The key is to not try to learn everything at once as well as have a definite path to walk through. Hence the 5 separate books.

I'm hoping at the end of this journey I will have accomplished all of these goals.

See you at the first Rakudo Core Fundraiser.

Perl 6: Seqs, Drugs, And Rock'n'Roll (Part 2)

Read this article on Perl6.Party

This is the second part in the series! Be sure you read Part I first where we discuss what Seqs are and how to .cache them.

Today, we'll take the Seq apart and see what's up in it; what drives it; and how to make it do exactly what we want.

PART II: That Iterated Quickly

The main piece that makes a Seq do its thing is an object that does the Iterator role. It's this object that knows how to generate the next value, whenever we try to pull a value from a Seq, or push all of its values somewhere, or simply discard all of the remaining values.

Keep in mind that you never need to use Iterator's methods directly, when making use of a Seq as a source of values. They are called indirectly under the hood in various Perl 6 constructs. The use case for calling those methods yourself is often the time when we're making an Iterator that's fed by another Iterator, as we'll see.

Pull my finger...

In its most basic form, an Iterator object needs to provide only one method: .pull-one

my $seq := Seq.new: class :: does Iterator {
    method pull-one {
        return $++ if $++ < 4;
        IterationEnd
    }
}.new;

.say for $seq;

# OUTPUT:
# 0
# 1
# 2
# 3

Above, we create a Seq using its .new method that expects an instantiated Iterator, for which we use an anonymous class that does the Iterator role and provides a single .pull-one method that uses a pair of anonymous state variables to generate 4 numbers, one per call, and then returns IterationEnd constant to signal the Iterator does not have any more values to produce.

The Iterator protocol forbids attempting to fetch more values from an Iterator once it generated the IterationEnd value, so your Iterator's methods may assume they'll never get called again past that point.

Meet the rest of the gang

The Iterator role defines several more methods, all of which are optional to implement, and most of which have some sort of default implementation. The extra methods are there for optimization purposes that let you take shortcuts depending on how the sequence is iterated over.

Let's build a Seq that hashes a bunch of data using Crypt::Bcryptmodule (run zef install Crypt::Bcrypt to install it). We'll start with the most basic Iterator that provides .pull-one method and then we'll optimize it to perform better in different circumstances.

use Crypt::Bcrypt;

sub hash-it (*@stuff) {
    Seq.new: class :: does Iterator {
        has @.stuff;
        method pull-one {
            @!stuff ?? bcrypt-hash @!stuff.shift, :15rounds
                    !! IterationEnd
        }
    }.new: :@stuff
}

my $hashes := hash-it <foo bar ber>;
for $hashes {
    say "Fetched value #{++$} {now - INIT now}";
    say "\t$_";
}

# OUTPUT:
# Fetched value #1 2.26035863
#     $2b$15$ZspycxXAHoiDpK99YuMWqeXUJX4XZ3cNNzTMwhfF8kEudqli.lSIa
# Fetched value #2 4.49311657
#     $2b$15$GiqWNgaaVbHABT6yBh7aAec0r5Vwl4AUPYmDqPlac.pK4RPOUNv1K
# Fetched value #3 6.71103435
#     $2b$15$zq0mf6Qv3Xv8oIDp686eYeTixCw1aF9/EqpV/bH2SohbbImXRSati

In the above program, we wrapped all the Seq making stuff inside a sub called hash-it. We slurp all the positional arguments given to that sub and instantiate a new Seq with an anonymous class as the Iterator. We use attribute @!stuff to store the stuff we need to hash. In the .pull-one method we check if we still have @!stuff to hash; if we do, we shift a value off @!stuff and hash it, using 15 rounds to make the hashing algo take some time. Lastly, we added a say statement to measure how long the program has been running for each iteration, using two now calls, one of which is run with the INIT phaser. From the output, we see it takes about 2.2 seconds to hash a single string.

Skipping breakfast

Using a for loop, is not the only way to use the Seq returned by our hashing routine. What if some user doesn't care about the first few hashes? For example, they could write a piece of code like this:

my $hash = hash-it(<foo bar ber>).skip(2).head;
say "Made hash {now - INIT now}";
say bcrypt-match 'ber', $hash;

# OUTPUT:
# Made hash 6.6813790
# True

We've used Crypt::Bcryptmodule's bcrypt-match routine to ensure the hash we got matches our third input string and it does, but look at the timing in the output. It took 6.7s to produce that single hash!

In fact, things will look the worse the more items the user tries to skip. If the user calls our hash-it with a ton of items and then tries to .skip the first 1,000,000 elements to get at the 1,000,001st hash, they'll be waiting for about 25 days for that single hash to be produced!!

The reason is our basic Iterator only knows how to .pull-one, so the skip operation still generates the hashes, just to discard them. Since the values our Iterator generates do not depend on previous values, we can implement one of the optimizing methods to skip iterations cheaply:

use Crypt::Bcrypt;

sub hash-it (*@stuff) {
    Seq.new: class :: does Iterator {
        has @.stuff;
        method pull-one {
            @!stuff ?? bcrypt-hash @!stuff.shift, :15rounds
                    !! IterationEnd
        }
        method skip-one {
            return False unless @!stuff;
            @!stuff.shift;
            True
        }
    }.new: :@stuff
}

my $hash = hash-it(<foo bar ber>).skip(2).head;
say "Made hash {now - INIT now}";
say bcrypt-match 'ber', $hash;

# OUTPUT:
# Made hash 2.2548012
# True

We added a .skip-one method to our Iterator that instead of hashing a value, simply discards it. It needs to return a truthy value, if it was able to skip a value (i.e. we had a value we'd otherwise generate in .pull-one, but we skipped it), or falsy value if there weren't any values to skip.

Now, the .skip method called on our Seq uses our new .skip-one method to cheaply skip through 2 items and then uses .pull-one to generate the third hash. Look at the timing now: 2.2s; the time it takes to generate a single hash.

However, we can kick it up a notch. While we won't notice a difference with our 3-item Seq, that user who was attempting to skip 1,000,000 items won't get the 2.2s time to generate the 1,000,000th hash. They would also have to wait for 1,000,000 calls to .skip-one and @!stuff.shift. To optimize skipping over a bunch of items, we can implement the .skip-at-least method (for brevity, just our Iterator class is shown):

class :: does Iterator {
    has @.stuff;
    method pull-one {
        @!stuff
            ?? bcrypt-hash( @!stuff.shift, :15rounds )
            !! IterationEnd
    }
    method skip-one {
        return False unless @!stuff;
        @!stuff.shift;
        True
    }
    method skip-at-least (Int \n) {
        n == @!stuff.splice: 0, n
    }
}

The .skip-at-least method takes an Int of items to skip. It should skip as many as it can, and return a truthy value if it was able to skip that many items, and falsy value if the number of skipped items was fewer. Now, the user who skips 1,000,000 items will only have to suffer through a single .splice call.

For the sake of completeness, there's another skipping method defined by Iterator: .skip-at-least-pull-one. It follows the same semantics as .skip-at-least, except with .pull-one semantics for return values. Its default implemention involves just calling those two methods, short-circuiting and returning IterationEnd if the .skip-at-least returned a falsy value, and that default implementation is very likely good enough for all Iterators. The method exists as a convenience for Iterator users who call methods on Iterators and (at the moment) it's not used in core Rakudo Perl 6 by any methods that can be called on users' Seqs.

A so, so count...

There are two more optimization methods—.bool-only and .count-only—that do not have a default implementation. The first one returns True or False, depending on whether there are still items that can be generated by the Iterator (True if yes). The second one returns the number of items the Iterator can still produce. Importantly these methods must be able to do that without exhausting the Iterator. In other words, after finding these methods implemented, the user of our Iterator can call them and afterwards should still be able to .pull-one all of the items, as if the methods were never called.

Let's make an Iterator that will take an Iterable and .rotate it once per iteration of our Iterator until its tail becomes its head. Basically, we want this:

.say for rotator 1, 2, 3, 4;

# OUTPUT:
# [2 3 4 1]
# [3 4 1 2]
# [4 1 2 3]

This iterator will serve our purpose to study the two Iterator methods. For a less "made-up" example, try to find implementations of iterators for combinations and permutations routines in Perl 6 compiler's source code.

Here's a sub that creates our Seq with our shiny Iterator along with some code that operates on it and some timings for different stages of the program:

sub rotator (*@stuff) {
    Seq.new: class :: does Iterator {
        has int $!n;
        has int $!steps = 1;
        has     @.stuff is required;

        submethod TWEAK { $!n = @!stuff − 1 }

        method pull-one {
            if $!n-- > 0 {
                LEAVE $!steps = 1;
                [@!stuff .= rotate: $!steps]
            }
            else {
                IterationEnd
            }
        }
        method skip-one {
            $!n > 0 or return False;
            $!n--; $!steps++;
            True
        }
        method skip-at-least (Int \n) {
            if $!n > all 0, n {
                $!steps += n;
                $!n     −= n;
                True
            }
            else {
                $!n = 0;
                False
            }
        }
    }.new: stuff => [@stuff]
}

my $rotations := rotator ^5000;

if $rotations {
    say "Time after getting Bool: {now - INIT now}";

    say "We got $rotations.elems() rotations!";
    say "Time after getting count: {now - INIT now}";

    say "Fetching last one...";
    say "Last one's first 5 elements are: $rotations.tail.head(5)";
    say "Time after getting last elem: {now - INIT now}";
}

# OUTPUT:
# Time after getting Bool: 0.0230339
# We got 4999 rotations!
# Time after getting count: 26.04481484
# Fetching last one...
# Last one's first 5 elements are: 4999 0 1 2 3
# Time after getting last elem: 26.0466234

First things first, let's take a look at what we're doing in our Iterator. We take an Iterable (in the sub call on line 37, we use a Range object out of which we can milk 5000 elements in this case), shallow-clone it (using [ ... ] operator) and keep that clone in @!stuff attribute of our Iterator. During object instantiation, we also save how many items @!stuff has in it into $!n attribute, inside the TWEAK submethod.

For each .pull-one of the Iterator, we .rotate our @!stuff attribute, storing the rotated result back in it, as well as making a shallow clone of it, which is what we return for the iteration.

We also already implemented the .skip-one and .skip-at-least optimization methods, where we use a private $!steps attribute to alter how many steps the next .pull-one will .rotate our @!stuff by. Whenever .pull-one is called, we simply reset $!steps to its default value of 1 using the LEAVE phaser.

Let's check out how this thing performs! We store our precious Seq in $rotations variable that we first check for truthiness, to see if it has any elements in it at all; then we tell the world how many rotations we can fish out of that Seq; lastly, we fetch the last element of the Seq and (for screen space reasons) print the first 5 elements of the last rotation.

All three steps—check .Bool, check .elems, and fetch last item with .tail are timed, and the results aren't that pretty. While .Bool took relatively quick to complete, the .elems call took ages (26s)! That's actually not all of the damage. Recall from PART I of this series that both .Bool and .elems cache the Seq unless special methods are implemented in the Iterator. This means that each of those rotations we made are still there in memory, using up space for nothing! What are we to do? Let's try implementing those special methods .Bool and .elems are looking for!

This only thing we need to change is to add two extra methods to our Iterator that determinine how many elements we can generate (.count-only) and whether we have any elements to generate (.bool-only):

method count-only { $!n     }
method bool-only  { $!n > 0 }

For the sake of completeness, here is our previous example, with these two methods added to our Iterator:

sub rotator (*@stuff) {
    Seq.new: class :: does Iterator {
        has int $!n;
        has int $!steps = 1;
        has     @.stuff is required;

        submethod TWEAK { $!n = @!stuff − 1 }

        method count-only { $!n     }
        method bool-only  { $!n > 0 }

        method pull-one {
            if $!n-- > 0 {
                LEAVE $!steps = 1;
                [@!stuff .= rotate: $!steps]
            }
            else {
                IterationEnd
            }
        }
        method skip-one {
            $!n > 0 or return False;
            $!n--; $!steps++;
            True
        }
        method skip-at-least (\n) {
            if $!n > all 0, n {
                $!steps += n;
                $!n     −= n;
                True
            }
            else {
                $!n = 0;
                False
            }
        }
    }.new: stuff => [@stuff]
}

my $rotations := rotator ^5000;

if $rotations {
    say "Time after getting Bool: {now - INIT now}";

    say "We got $rotations.elems() rotations!";
    say "Time after getting count: {now - INIT now}";

    say "Fetching last one...";
    say "Last one's first 5 elements are: $rotations.tail.head(5)";
    say "Time after getting last elem: {now - INIT now}";
}

# OUTPUT:
# Time after getting Bool: 0.0087576
# We got 4999 rotations!
# Time after getting count: 0.00993624
# Fetching last one...
# Last one's first 5 elements are: 4999 0 1 2 3
# Time after getting last elem: 0.0149863

The code is nearly identical, but look at those sweet, sweet timings! Our entire program runs about 1,733 times faster because our Seq can figure out if and how many elements it has without having to iterate or rotate anything. The .tail call sees our optimization (side note: that's actually very recent) and it too doesn't have to iterate over anything and can just use our .skip-at-least optimization to skip to the end. And last but not least, our Seq is no longer being cached, so the only things kept around in memory are the things we care about. It's a huge win-win-win for very little extra code.

But wait... there's more!

Push it real good...

The Seqs we looked at so far did heavy work: each generated value took a relatively long time to generate. However, Seqs are quite versatile and at times you'll find that generation of a value is cheaper than calling .pull-one and storing that value somewhere. For cases like that, there're a few more methods we can implement to make our Seq perform better.

For the next example, we'll stick with the basics. Our Iterator will generate a sequence of positive even numbers up to the wanted limit. Here's what the call to the sub that makes our Seq looks like:

say evens-up-to 20; # OUTPUT: (2 4 6 8 10 12 14 16 18)

And here's the all of the code for it. The particular operation we'll be doing is storing all the values in an Array, by assigning to it:

sub evens-up-to {
    Seq.new: class :: does Iterator {
        has int $!n = 0;
        has int $.limit is required;
        method pull-one { ($!n += 2) < $!limit ?? $!n !! IterationEnd }
    }.new: :$^limit
}

my @a = evens-up-to 1_700_000;

say now - INIT now; # OUTPUT: 1.00765440

For a limit of 1.7 million, the code takes around a second to run. However, all we do in our Iterator is add some numbers together, so a lot of the time is likely lost in .pull-oneing the values and adding them to the Array, one by one.

In cases like this, implementing a custom .push-all method in our Iterator can help. The method receives one argument that is a reification target. We're pretty close to bare "metal" now, so we can't do anything fancy with the reification target object other than call .push method on it with a single value to add to the target. The .push-all always returns IterationEnd, since it exhausts the Iterator, so we'll just pop that value right into the return value of the method's Signature:

sub evens-up-to {
    Seq.new: class :: does Iterator {
        has int $!n = 0;
        has int $.limit is required;
        method pull-one {
            ($!n += 2) < $!limit ?? $!n !! IterationEnd
        }
        method push-all (\target --> IterationEnd) {
            target.push: $!n while ($!n += 2) < $!limit;
        }
    }.new: :$^limit
}

my @a = evens-up-to 1_700_000;
say now - INIT now; # OUTPUT: 0.91364949

Our program is now 10% faster; not a lot. However, since we're doing all the work in .push-all now, we no longer need to deal with state inside the method's body, so we can shave off a bit of time by using lexical variables instead of accessing object's attributes all the time. We'll make them use native int types for even more speed. Also, (at least currently), the += meta operator is more expensive than a simple assignment and a regular +; since we're trying to squeeze every last bit of juice here, let's take advantage of that as well. So what we have now is this:

sub evens-up-to {
    Seq.new: class :: does Iterator {
        has int $!n = 0;
        has int $.limit is required;
        method pull-one {
            ($!n += 2) < $!limit ?? $!n !! IterationEnd
        }
        method push-all (\target --> IterationEnd) {
            my int $limit = $!limit;
            my int $n     = $!n;
            target.push: $n while ($n = $n + 2) < $limit;
            $!n = $n;
        }
    }.new: :$^limit
}

my @a = evens-up-to 1_700_000;
say now - INIT now; # OUTPUT: 0.6688109

There we go. Now our program is 1.5 times faster than the original, thanks to .push-all. The gain isn't as dramatic as we what saw with other methods, but can come in quite handy when you need it.

There are a few more .push-* methods you can implement to, for example, do something special when your Seq is used in codes like...

for $your-seq -> $a, $b, $c { ... }

...where the Iterator would be asked to .push-exactly three items. The idea behind them is similar to .push-all: you push stuff onto the reification target. Their utility and performance gains are ever smaller, useful only in particular situations, so I won't be covering them.

It's worth noting the .push-all can be used only with Iterators that are not lazy, since... well... it expects you to push all the items. And what exactly are lazy Iterators? I'm so glad you asked!

A quick brown fox jumped over the lazy Seq

Let's pare down our previous Seq that generates even numbers down to the basics. Let's make it generate an infinite list of even numbers, using an anonymous state variable:

sub evens {
    Seq.new: class :: does Iterator {
        method pull-one { $ += 2 }
    }.new
}

put evens

Since the list is infinite, it'd take us an infinite time to fetch them all. So what exactly happens when we run the code above? It... quite predictably hangs when the put routine is called; it sits and patiently waits for our infinite Seq to complete. The same issue occurs when trying to assign our seq to a @-sigiled variable:

my @evens = evens # hangs

Or even when trying to pass our Seq to a sub with a slurpy parameter_Parameters):

sub meows (*@evens) { say 'Got some evens!' }
meows evens # hangs

That's quite an annoying problem. Fortunately, there's a very easy solution for it. But first, a minor detour to the land of naming clarification!

A rose by any other name would laze as sweet

In Perl 6 some things are or can be made "lazy". While it evokes the concept of on-demand or "lazy" evaluation, which is ubiquitous in Perl 6, things that are lazy in Perl 6 aren't just about that. If something is-lazy, it means it always wants to be evaluated lazily, fetching only as many items as needed, even in "mostly lazy" Perl 6 constructs that would otherwise eagerly consume even from sources that do on-demand generation.

For example, a sequence of lines read from a file would want to be lazy, as reading them all in at once has the potential to use up all the RAM. An infinite sequence would also want to be is-lazy because an eager evaluation would cause it to hang, as the sequence never completes.

So a thing that is-lazy in Perl 6 can be thought of as being infinite. Sometimes it actually will be infinite, but even if it isn't, it being lazy means it has similar consequences if used eagerly (too much CPU time used, too much RAM, etc).


Now back to our infinite list of even numbers. It sounds like all we have to do is make our Seq lazy and we do that by implementing .is-lazy method on our Iterator that simply returns True:

sub evens {
    Seq.new: class :: does Iterator {
        method pull-one { $ += 2 }
        method is-lazy (--> True) {}
    }.new
}

sub meows (*@evens) { say 'Got some evens!' }

put         evens; # OUTPUT: ...
my @evens = evens; # doesn't hang
meows       evens; # OUTPUT: Got some evens!

The put routine now detects its dealing with something terribly long and just outputs some dots. Assignment to Array no longer hangs (and will instead reify on demand). And the call to a slurpy doesn't hang either and will also reify on demand.

There's one more Iterator optimization method left that we should discuss...

A Sinking Ship

Perl 6 has sink context, similar to "void" context in other languages, which means a value is being discarded:

42;

# OUTPUT:
# WARNINGS for ...:
# Useless use of constant integer 42 in sink context (line 1)

The constant 42 in the above program is in sink context—its value isn't used by anything—and since it's nearly pointless to have it like that, the compiler warns about it.

Not all sinkage is bad however and sometimes you may find that gorgeous Seq on which you worked so hard is ruthlessly being sunk by the user! Let's take a look at what happens when we sink one of our previous examples, the Seq that generates up to limit even numbers:

sub evens-up-to {
    Seq.new: class :: does Iterator {
        has int $!n = 0;
        has int $.limit is required;
        method pull-one {
            ($!n += 2) < $!limit ?? $!n !! IterationEnd
        }
    }.new: :$^limit
}

evens-up-to 5_000_000; # sink our Seq

say now - INIT now; # OUTPUT: 5.87409072

Ouch! Iterating our Seq has no side-effects outside of the Iterator that it uses, which means it took the program almost six seconds to do absolutely nothing.

We can remedy the situation by implementing our own .sink-all method. Its default implementation .pull-ones until the end of the Seq (since Seqs may have useful side effects), which is not what we want for our Seq. So let's implement a .sink-all that does nothing!

sub evens-up-to {
    Seq.new: class :: does Iterator {
        has int $!n = 0;
        has int $.limit is required;
        method pull-one {
            ($!n += 2) < $!limit ?? $!n !! IterationEnd
        }
        method sink-all(--> IterationEnd) {}
    }.new: :$^limit
}

evens-up-to 5_000_000; # sink our Seq

say now - INIT now; # OUTPUT: 0.0038638

We added a single line of code and made our program 1,520 times faster—the perfect speed up for a program that does nothing!

However, doing nothing is not the only thing .sink-all is good for. Use it for clean up that would usually be done at the end of iteration (e.g. closing a file handle the Iterator was using). Or simply set the state of the system to what it would be at the end of the iteration (e.g. .seek a file handle to the end, for sunk Seq that produces lines from it). Or, as an alternative idea, how about warning the user their code might contain an error:

sub evens-up-to {
    Seq.new: class :: does Iterator {
        has int $!n = 0;
        has int $.limit is required;
        method pull-one {
            ($!n += 2) < $!limit ?? $!n !! IterationEnd
        }
        method sink-all(--> IterationEnd) {
            warn "Oh noes! Looks like you sunk all the evens!\n"
                ~ 'Why did you make them in the first place?'
        }
    }.new: :$^limit
}

evens-up-to 5_000_000; # sink our Seq

# OUTPUT:
# Oh noes! Looks like you sunk all the evens!
# Why did you make them in the first place?
# ...

That concludes our discussion on optimizing your Iterators. Now, let's talk about using Iterators others have made.

It's a marathon, not a sprint

With all the juicy knowledge about Iterators and Seqs we now possess, we can probably see how this piece of code manages to work without hanging, despite being given an infinite Range of numbers:

.say for ^∞ .grep(*.is-prime).map(* ~ ' is a prime number').head: 5;

# OUTPUT:
# 2 is a prime number
# 3 is a prime number
# 5 is a prime number
# 7 is a prime number
# 11 is a prime number

The infinite Range probably is-lazy. That .grep probably .pull-ones until it finds a prime number. The .map .pull-ones each of the .grep's values and modifies them, and .head allows at most 5 values to be .pull-oned from it.

In short what we have here is a pipeline of Seqs and Iterators where the Iterator of the next Seq is based on the Iterator of the previous one. For our study purposes, let's cook up a Seq of our own that combines all of the steps above:

sub first-five-primes (*@numbers) {
    Seq.new: class :: does Iterator {
        has     $.iter;
        has int $!produced = 0;
        method pull-one {
            $!produced++ == 5 and return IterationEnd;
            loop {
                my $value := $!iter.pull-one;
                return IterationEnd if $value =:= IterationEnd;
                return "$value is a prime number" if $value.is-prime;
            }
        }
    }.new: iter => @numbers.iterator
}

.say for first-five-primes ^∞;

# OUTPUT:
# 2 is a prime number
# 3 is a prime number
# 5 is a prime number
# 7 is a prime number
# 11 is a prime number

Our sub slurps up_Parameters) its positional arguments and then calls .iterator method on the @numbers Iterable. This method is available on all Perl 6 objects and will let us interface with the object using Iterator methods directly.

We save the @numbers's Iterator in one of the attributes of our Iterator as well as create another attribute to keep track of how many items we produced. In the .pull-one method, we first check whether we already produced the 5 items we need to produce, and if not, we drop into a loop that calls .pull-one on the other Iterator, the one we got from @numbers Array.

We recently learned that if the Iterator does not have any more values for us, it will return the IterationEnd constant. A constant whose job is to signal the end of iteration is finicky to deal with, as you can imagine. To detect it, we need to ensure we use the binding (:=), not the assignment (=) operator, when storing the value we get from .pull-one in a variable. This is because pretty much only the container identity (=:=) operator will accept such a monstrosity, so we can't stuff the value we .pull-one into just any container we please.

In our example program, if we do find that we received IterationEnd from the source Iterator, we simply return it to indicate we're done. If not, we repeat the process until we find a prime number, which we then put into our desired string and that's what we return from our .pull-one.

All the rest of the Iterator methods we've learned about can be called on the source Iterator in a similar fashion as we called .pull-one in our example.

Conclusion

Today, we've learned a whole ton of stuff! We now know that Seqs are powered by Iterator objects and we can make custom iterators that generate any variety of values we can dream about.

The most basic Iterator has only .pull-one method that generates a single value and returns IterationEnd when it has no more values to produce. It's not permitted to call .pull-one again, once it generated IterationEnd and we can write our .pull-one methods with the expectation that will never happen.

There are plenty of optimization opportunities a custom Iterator can take advantage of. If it can cheaply skip through items, it can implement .skip-one or .skip-at-least methods. If it can know how many items it'll produce, it can implement .bool-only and .count-only methods that can avoid a ton of work and memory use when only certain values of a Seq are needed. And for squeezing the very last bit of performance, you can take advantage of .push-all and other .push-* methods that let you push values onto the target directly.

When your Iterator .is-lazy, things will treat it with extra care and won't try to fetch all of the items at once. And we can use the .sink-all method to avoid work or warn the user of potential mistakes in their code, when our Seq is being sunk.

Lastly, since we know how to make Iterators and what their methods do, we can make use of Iterators coming from other sources and call methods on them directly, manipulating them just how we want to.

We now have all the tools to work with Seq objects in Perl 6. In the PART III of this series, we'll learn how to compactify all of that knowledge and skillfully build Seqs with just a line or two of code, using the sequence operator.

Stay tuned!

-Ofun

Perl 6: Seqs, Drugs, And Rock'n'Roll

Read this article on Perl6.Party

I vividly recall my first steps in Perl 6 were just a couple of months before the first stable release of the language in December 2015. Around that time, Larry Wall was making a presentation and showed a neat feature—the sequence operator—and it got me amazed about just how powerful the language is:

# First 12 even numbers:
say (2, 4 … ∞)[^12];      # OUTPUT: (2 4 6 8 10 12 14 16 18 20 22 24)

# First 10 powers of 2:
say (2, 2², 2³ … ∞)[^10]; # OUTPUT: (2 4 8 16 32 64 128 256 512 1024)

# First 13 Fibonacci numbers:
say (1, 1, *+* … ∞)[^13]; # OUTPUT: (1 1 2 3 5 8 13 21 34 55 89 144 233)

The ellipsis () is the sequence operator and the stuff it makes is the Seq object. And now, a year and a half after Perl 6's first release, I hope to pass on my amazement to a new batch of future Perl 6 programmers.

This is a 3-part series. In PART I of this article we'll talk about what Seq s are and how to make them without the sequence operator. In PART II, we'll look at the thing-behind-the-curtain of Seq's: the Iterator type and how to make Seqs from our own Iterators. Lastly, in PART III, we'll examine the sequence operator in all of its glory.

Note: I will be using all sorts of fancy Unicode operators and symbols in this article. If you don't like them, consult with the Texas Equivalents page for the equivalent ASCII-only way to type those elements.

PART I: What the Seq is all this about?

The Seq stands for Sequence and the Seq object provides a one-shot way to iterate over a sequence of stuff. New values can be generated on demand—in fact, it's perfectly possible to create infinite sequences—and already-generated values are discarded, never to be seen again, although, there's a way to cache them, as we'll see.

Sequences are driven by Iterator objects that are responsible for generating values. However, in many cases you don't have to create Iterators directly or use their methods while iterating a Seq. There are several ways to make a Seq and in this section, we'll talk about gather/take construct.

I gather you'll take us to...

The gather statement and take routine are similar to "generators" and "yield" statement in some other languages:

my $seq-full-of-sunshine := gather {
    say  'And nobody cries';
    say  'there’s only butterflies';

    take 'me away';
    say  'A secret place';
    say  'A sweet escape';

    take 'meee awaaay';
    say  'To better days'    ;

    take 'MEEE AWAAAAYYYY';
    say  'A hiding place';
}

Above, we have a code block with lines of song lyrics, some of which we say (print to the screen) and others we take (to be gathered). Just like, .say can be used as either a method or a subroutine, so you can use .take as a method or subroutine, there's no real difference; merely convenience.

Now, let's iterate over $seq-full-of-sunshine and watch the output:

for $seq-full-of-sunshine {
    ENTER say '▬▬▶ Entering';
    LEAVE say '◀▬▬ Leaving';

    say "❚❚ $_";
}

# OUTPUT:
# And nobody cries
# there’s only butterflies
# ▬▬▶ Entering
# ❚❚ me away
# ◀▬▬ Leaving
# A secret place
# A sweet escape
# ▬▬▶ Entering
# ❚❚ meee awaaay
# ◀▬▬ Leaving
# To better days
# ▬▬▶ Entering
# ❚❚ MEEE AWAAAAYYYY
# ◀▬▬ Leaving
# A hiding place

Notice how the say statements we had inside the gather statement didn't actualy get executed until we needed to iterate over a value that take routines took after those particular say lines. The block got stopped and then continued only when more values from the Seq were requested. The last say call didn't have any more takes after it, and it got executed when the iterator was asked for more values after the last take.

That's exceptional!

The take routine works by throwing a CX::Take control exception that will percolate up the call stack until something takes care of it. This means you can feed a gather not just from an immediate block, but from a bunch of different sources, such as routine calls:

multi what's-that (42)                     { take 'The Answer'            }
multi what's-that (Int $ where *.is-prime) { take 'Tis a prime!'          }
multi what's-that (Numeric)                { take 'Some kind of a number' }

multi what's-that   { how-good-is $^it                   }
sub how-good-is ($) { take rand > ½ ?? 'Tis OK' !! 'Eww' }

my $seq := gather map &what's-that, 1, 31337, 42, 'meows';

.say for $seq;

# OUTPUT:
# Some kind of a number
# Tis a prime!
# The Answer
# Eww

Once again, we iterated over our new Seq with a for loop, and you can see that take called from different multies and even nested sub calls still delivered the value to our gather successfully:

The only limitation is you can't gather takes done in another Promise or in code manually cued in the scheduler:

gather await start take 42;
# OUTPUT:
# Tried to get the result of a broken Promise
#   in block <unit> at test.p6 line 2
#
# Original exception:
#     take without gather

gather $*SCHEDULER.cue: { take 42 }
await Promise.in: 2;
# OUTPUT: Unhandled exception: take without gather

However, nothing's stopping you from using a Channel to proxy your data to be taken in a react block.

my Channel $chan .= new;
my $promise = start gather react whenever $chan { .take }

say "Sending stuff to Channel to gather...";
await start {
    $chan.send: $_ for <a b c>;
    $chan.close;
}
dd await $promise;

# OUTPUT:
# Sending stuff to Channel to gather...
# ("a", "b", "c").Seq

Or gathering takes from within a Supply:

my $supply = supply {
    take 42;
    emit 'Took 42!';
}

my $x := gather react whenever $supply { .say }
say $x;

# OUTPUT: Took 42!
# (42)

Stash into the cache

I mentioned earlier that Seqs are one-shot Iterables that can be iterated only once. So what exactly happens when we try to iterate them the second time?

my $seq := gather take 42;
.say for $seq;
.say for $seq;

# OUTPUT:
# 42
# This Seq has already been iterated, and its values consumed
# (you might solve this by adding .cache on usages of the Seq, or
# by assigning the Seq into an array)

A X::Seq::Consumed exception gets thrown. In fact, Seqs do not even do the Positional role, which is why we didn't use the @ sigil that type- checks for Positional on the variables we stored Seqs in.

The Seq is deemed consumed whenever something asks it for its Iterator after another thing grabbed it, like the for loop would. For example, even if in the first for loop above we would've iterated over just 1 item, we wouldn't be able to resume taking more items in the next for loop, as it'd try to ask for the Seq's iterator that was already taken by the first for loop.

As you can imagine, having Seqs always be one-shot would be somewhat of a pain in the butt. A lot of times you can afford to keep the entire sequence around, which is the price for being able to access its values more than once, and that's precisely what the Seq.cachemethod does:

my $seq := gather { take 42; take 70 };
$seq.cache;

.say for $seq;
.say for $seq;

# OUTPUT:
# 42
# 70
# 42
# 70

As long as you call .cache before you fetch the first item of the Seq, you're good to go iterating over it until the heat death of the Universe (or until its cache noms all of your RAM). However, often you do not even need to call .cache yourself.

Many methods will automatically .cache the Seq for you:

There's one more nicety with Seqs losing their one-shotness that you may see refered to as PositionalBindFailover. It's a role that indicates to the parameter binder that the type can still be converted into a Positional, even when it doesn't do Positional role. In plain English, it means you can do this:

sub foo (@pos) { say @pos[1, 3, 5] }

my $seq := 2, 4 … ∞;
foo $seq; # OUTPUT: (4 8 12)

We have a sub that expects a Positional argument and we give it a Seq which isn't Positional, yet it all works out, because the binder .caches our Seq and uses the List the .cache method returns to be the Positional to be used, thanks to it doing the PositionalBindFailover role.

Last, but not least, if you don't care about all of your Seq's values being generated and cached right there and then, you can simply assign it to a @ sigiled variable, which will reify the Seq and store it as an Array:

my @stuff = gather {
    take 42;
    say "meow";
    take 70;
}

say "Starting to iterate:";
.say for @stuff;

# OUTPUT:
# meow
# Starting to iterate:
# 42
# 70

From the output, we can see say "meow" was executed on assignment to @stuff and not when we actually iterated over the value in the for loop.

Conclusion

In Perl 6, Seqs are one-shot Iterables that don't keep their values around, which makes them very useful for iterating over huge, or even infinite, sequences. However, it's perfectly possible to cache Seq values and re-use them, if that is needed. In fact, many of the Seq's methods will automatically cache the Seq for you.

There are several ways to create Seqs, one of which is to use the gather and take where a gather block will stop its execution and continue it only when more values are needed.

In parts II and III, we'll look at other, more exciting, ways of creating Seqs. Stay tuned!

-Ofun

Perl 6 Release Quality Assurance: Full Ecosystem Toaster

Read this article on Perl6.Party

As some recall, Rakudo's 2017.04 release was somewhat of a trainwreck. It was clear the quality assurance of releases needed to be kicked up a notch. So today, I'll talk about what progress we've made in that area.

Define The Problem

A particular problem that plagued the 2017.04 release were big changes and refactors made in the compiler that passed all the 150,000+ stresstests, however still caused issues in some ecosystem modules and users' code.

The upcoming 2017.06 has many, many more big changes:

  • IO::ArgFiles were entirely replaced with the new IO::CatHandle implementation
  • IO::Socket got a refactor and sync sockets no longer use libuv
  • IO::Handle got a refactor with encoding and sync IO no longer uses libuv
  • Sets/Bags/Mixes got optimization polish and op semantics finalizations
  • Proc was refactored to be in terms of Proc::Async

The IO and Proc stuff is especially impactful, as it affects precomp and module loading as well. Merely passing stresstests just wouldn't give me enough of peace of mind of a solid release. It was time to extend the testing.

Going All In

The good news is I didn't actually have to write any new tests. With 836 modules in the Perl 6 ecosystem, the tests were already there for the taking. Best of all, they were mostly written without bias due to implementation knowledge of core code, as well as have personal style variations from hundreds of different coders. This is all perfect for testing for any regressions of core code. The only problem is running all that.

While there's a budding effort to get CPANTesters to smoke Perl 6 dists, it's not quite the data I need. I need to smoke a whole ton of modules on a particular pre-release commit, while also smoking them on a previous release on the same box, eliminating setup issues that might contribute to failures, as well as ensuring the results were for the same versions of modules.

My first crude attempt involved firing up a 32-core Google Compute Engine VM and writing a 60-line script that launched 836 Proc::Asyncs—one for each module.

Other than chewing through 125 GB of RAM with a single Perl 6 program, the experiment didn't yield any useful data. Each module had to wait for locks, before being installed, and all the Procs were asking zef to install to the same location, so dependency handling was iffy. I needed a more refined solution...

Procs, Kernels, and Murder

So, I started to polish my code. First, I wrote Proc::Q module that let me queue up a bunch of Procs, and scale the number of them running at the same time, based on the number of cores the box had. Supply.throttle core feature made the job a piece of cake.

However, some modules are naughty or broken and I needed a way to kill Procs that take too long to run. Alas, I discovered that Proc::Async.kill had a bug in it, where trying to simultaneously kill a bunch of Procs was failing. After some digging I found out the cause was $*KERNEL.signal method the .kill was using isn't actually thread safe and the bug was due to a data race in initialization of the signal table.

After refactoring Kernel.signal, and fixing Proc::Async.kill, I released Proc::Q module—my first module to require (at the time) the bleedest of bleeding edges: a HEAD commit.

Going Atomic

After cooking up boilerplate DB and Proc::Q code, I was ready to toast the ecosystem. However, it appeared zef wasn't designed, or at least well-tested, in scenarious where up to 40 instances were running module installations simultaneously. I was getting JSON errors from reading ecosystem JSON, broken cache files (due to lack of file locking), and false positives in installations because modules claimed they were already installed.

I initially attempted to solve the JSON errors by looking at an Issue in the ecosystem repo about the updater script not writing atomically. However, even after fixing the updater script, I was still getting invalid JSON errors from zef when reading ecosystem data.

It might be due to something in zef, but instead of investigating it further, I followed ugexe++'s advice and told zef not to fetch ecosystem in each Proc. The broken cache issues were similarly eliminated by disabling caching support. And the false positives were eliminated telling each zef instance to install the tested module into a separate location.

The final solution involved programatically editing zef's config file before a toast run to disable auto-updates of CPAN and p6c ecosystem data, and then in individual Procs zef module install command ended up being:

«zef --/cached --debug install "$module" "--install-to=inst#$where"»

Where $where is a per-module, per-rakudo-commit location. The final issue was floppy test runs, which I resolved by re-testing failed modules one more time, to see if the new run succeeds.

Time is Everything

The toasting of the entire ecosystem on HEAD and 2017.05 releases took about three hours on a 24-core VM, while being unattended. While watching over it and killing the few hanging modules at the end without waiting for them to time out makes a single-commit run take about 65 minutes.

I also did a toast run on a 64-core VM...

Overall, the run took me 50 minutes, and I had to manually kill some modules' tests. However, looking at CPU utilization charts, it seems the run sat idle for dozens of minutes before I came along to kill stuff:

So I think after some polish of avoiding hanging modules and figuring out why (apparently) Proc::Async.kill still doesn't kill everything, the runs can be entirely automated and a single run can be completed in about 20-30 minutes.

This means that even with last-minute big changes pushed to Rakudo, I can still toast the entire ecosystem reasonably fast, detect any potential regressions, fix them, and re-test again.

Reeling In The Catch

The Toaster database is available for viewing at toast.perl6.party. As more commits get toasted, they get added to the database. I plan to clear them out after each release.

The toasting runs I did so far weren't just a chance to play with powerful hardware. The very first issue was detected when toasting Clifford module.

The issue was to do with Lists of Pairs with same keys coerced into a MixHash, when the final accumulative weight was zero. The issue was introduced on June 7th and it took me about an hour of digging through the module's guts to find it. Considering it's quite an edge case, I imagine without the toaster runs it would take a lot longer to identify this bug. lizmat++ squashed this bug hours after identification and it never made it into any releases.

The other issue detected by toasting had to do with the VM-backed decoder serialization introduced during IO refactor and jnthn++ fixed it a day after detection. One more bug had to do with Proc refactor making Proc not synchronous-enough. It was mercilessly squashed, while fixing a couple of longstanding issues with Proc.

All of these issues weren't detected by the 150,000+ tests in the testsuite and while an argument can be made that the tests are sparse in places, there's no doubt the Toaster has paid off for the effort in making it by catching bugs that might've otherwise made it into the release.

The Future

The future plans for the Toaster would be first to make it toast on more platforms, like Windows and MacOS. Eventually, I hope to make toast runs continuous, on less-powerful VMs that are entirely automated. An IRC bot would watch for any failures and report them to the dev channel.

Conclusion

The ecosystem Toaster lets core devs test a Rakudo commit on hundreds of software pieces, made by hundreds of different developers, all within a single hour. During its short existence, the Toaster already found issues with ecosystem infrastructure, highly-multi-threaded Perl 6 programs, as well as detected regressions and new bugs that we were able to fix before the release.

The extra testing lets core devs deliver higher-quality releases, which makes Perl 6 more trustworthy to use in production-quality software. The future will see the Toaster improved to test on a wider range of systems, as well as being automated for continued extended testing.

And most importantly, the Toaster makes it possible for any Perl 6 programmer to help core development of Perl 6, by simply publishing a module.

-Ofun

About Zoffix Znet

user-pic I blog about Perl.