Read this article on Rakudo.Party

If I were to pick the most ubiquitous construct in the Perl 6 programming language, it'd most definitely be the colonpair. Hash constructors, named arguments and parameters, adverbs, and regex modifiers—all involve the colonpair. It's not surprising that with such breadth there would be many shortcuts when it comes to constructing colonpairs.

Today, we'll learn about all of those! Doing so will have us looking at the simplest as well as some of the more advanced language constructs, so if parts of this article make you scratch your head, don't worry—you don't have to learn all of it at once!

PART I: Creation

Colonwhaaaa?

The colonpair gets its name from (usually) being a Pair object constructor and (usually) having a colon in it. Here are some examples of colonpairs:

:foo,
:$bar,
:meow<moo>,
heh => hah

The last one doesn't have a colon in it, but since it's basically the same thing as other colonpairs, I personally consider it a colonpair as well.

We can see the colonpairs make Pair objects by dumping their .^namemethodname):

say :foo.^name; # OUTPUT: «Pair␤»

However, when used in argument lists, the colonpairs are specially handled to represent named arguments. We'll get to that part later in the article.

The Shortcuts

Here's a mostly-complete list of available ways to write a colonpair you can glance over before we dive in. I know, it looks like a huge list, but that's why we're reading this article—to learn the general patterns that make up all of these permutations.

# Standard, take-any-type, non-shortcut form
:nd(2).say;             # OUTPUT: «nd => 2␤»
:foo('foo', 'bar').say; # OUTPUT: «foo => (foo bar)␤»
:foo( %(:42a, :foo<a b c>) ).say;
# OUTPUT: «foo => {a => 42, foo => (a b c)}␤»

# Can use fat-arrow notation too:
# (parentheses around them are here just for the .say call)
(nd => 2).say; # OUTPUT: «nd => 2␤»
(foo => ('foo', 'bar') ).say; # OUTPUT: «foo => (foo bar)␤»
(foo => %(:42a, :foo<a b c>) ).say;
# OUTPUT: «foo => {a => 42, foo => (a b c)}␤»

# Booleans
:foo .say; # OUTPUT: «foo => True␤»
:!foo.say; # OUTPUT: «foo => False␤»

# Unsigned integers:
:2nd   .say; # OUTPUT: «nd => 2␤»
:1000th.say; # OUTPUT: «th => 1000␤»

# Strings and Allomorphs (stings that look like numbers are Str + numeric type)
:foo<bar>      .say; # OUTPUT: «foo => bar␤»
:bar<42.5>     .say; # OUTPUT: «bar => 42.5␤»
:bar<42.5>.perl.say; # OUTPUT: «:bar(RatStr.new(42.5, "42.5"))␤»

# Positionals
:foo['foo', 42.5] .say; # A mutable Array:   OUTPUT: «foo => [foo 42.5]␤»
:foo<foo bar 42.5>.say; # An immutable List: OUTPUT: «foo => (foo bar 42.5)␤»
# angled brackets give you allomorphs!

# Callables
:foo{ say "Hello, World!" }.say;
# OUTPUT: «foo => -> ;; $_? is raw { #`(Block|82978224) ... }␤»

# Hashes; keep 'em simple so it doesn't get parsed as a Callable
:foo{ :42a, :foo<a b c> }.say; # OUTPUT: «foo => {a => 42, foo => (a b c)}␤»

# Name and value from variable
:$foo;  # same as :foo($foo)
:$*foo; # same as :foo($*foo)
:$?foo; # same as :foo($?foo)
:$.foo; # same as :foo($.foo)
:$!foo; # same as :foo($!foo)
:@foo;  # same as :foo(@foo)
:@*foo; # same as :foo(@*foo)
:@?foo; # same as :foo(@?foo)
:@.foo; # same as :foo(@.foo)
:@!foo; # same as :foo(@!foo)
:%foo;  # same as :foo(%foo)
:%*foo; # same as :foo(%*foo)
:%?foo; # same as :foo(%?foo)
:%.foo; # same as :foo(%.foo)
:%!foo; # same as :foo(%!foo)
:&foo;  # same as :foo(&foo)
:&*foo; # same as :foo(&*foo)
:&?foo; # same as :foo(&?foo)
:&.foo; # same as :foo(&.foo)
:&!foo; # same as :foo(&!foo)

Let's break these up and take a closer look!

Standard, Take-any-Type, Non-Shortcut Form

The "standard" form of the colonpair consists of a colon (:), a valid term that functions as the .key of the created Pair object, and then a set of parentheses inside of which is the expression with the .value for the Pair:

:nd(2).say;                       # OUTPUT: «nd => 2␤»
:foo('foo', 'bar').say;           # OUTPUT: «foo => (foo bar)␤»
:foo( %(:42a, :foo<a b c>) ).say;
# OUTPUT: «foo => {a => 42, foo => (a b c)}␤»

As long as the key is a valid identifier, all other forms of colonpairs can be written using this way. And for non-valid identifiers, you can simply use the .new method—Pair.new('the key','value')—or the "fat arrow" syntax.

Fat Arrow Syntax

If you ever used Perl 5, you need no introductions to this syntax: you write the key—which will get auto-quoted if it's a valid identifier, so in those cases you can omit the quotes—then you write => and then you write the value. The quotes around the key are required if the key is not a valid identifier and the fat arrow is the only operator-involved syntax that will let you construct Pairs with such keys:

# (outer parentheses are here just for the .say call)
(nd => 2).say; # OUTPUT: «nd => 2␤»
(foo => ('foo', 'bar') ).say; # OUTPUT: «foo => (foo bar)␤»
(foo => %(:42a, :foo<a b c>) ).say;
# OUTPUT: «foo => {a => 42, foo => (a b c)}␤»
("the key" => "the value").say; # OUTPUT: «the key => the value␤»

There are some extra rules with how this form behaves in argument lists as well as sigilless variables and constants, which we'll see later in the article.

Boolean Shortcut

Now we start getting into shortcuts! What would the most common use of named parameters be? Probably, to specify boolean flags.

It'd be pretty annoying to always have to write those as :foo(True), so there's a shortcut: simply omit the value entirely, and if you want :foo(False), omit the value and put the negation operator) right after the colon:

# Booleans
:foo .say; # OUTPUT: «foo => True␤»
:!foo.say; # OUTPUT: «foo => False␤»

# Equivalent calls:
some-sub :foo :!bar :ber;
some-sub foo => True, bar => False, ber => True;

The shortcut form is a lot shorter. This is also the form you may see in adverbs and regex syntax, such as the :g adverb on the m// quoter and :s/:!s significant whitespace modifier inside the regex:

say "a b c def g h i" ~~ m:g/:s \S \S [:!s \S \s+ \S]/;
# OUTPUT: «(｢a b c d｣ ｢f g h i｣)␤»

Here's also another trick from my personal bag: since Bool type is an Int, you can use boolean shortcuts to specify Int values 1 and 0:

# set `batch` to `1`:
^4 .race(:batch).map: { sleep 1 };
say now - ENTER now; # OUTPUT: «1.144883␤»

However, for clarity you may wish to use unsigned integer colonpair shortcut instead, which isn't much longer.

Unsigned Integer Shortcut

The Perl 6 programming language lets you grab an nth match when you're matching stuff with a regex:

say "first second third" ~~ m:3rd/\S+/;
# OUTPUT: «｢third｣␤»

As you can probably surmise by now, the :3rd after the m in m// quoter is the adverb, written as a colonpair in unsigned integer shortcut. This form consist of a colon and the name of the key with unquoted unsigned integer value placed between them. No signs, no decimal dots, and not even underscore separators between digits are permitted.

The primary use of this shortcut is for things with ordinal suffixes like :1st, :2nd, :9th, etc. It offers great readability there, but personally I have no reservations about using this syntax for all unsigned integer values, regardless of what the name of the key is. It feels slightly offcolour when you first encounter such syntax, but it quickly grows on you:

some-sub :1st :2nd :3rd :42foo :100bar;
^4 .race(:1batch).map: { sleep 1 };

Hash/Array/Callable Shortcuts

Using standard colonpair format you may notice some forms are too parentheses-heavy:

:foo(<42>)                 # value is an IntStr allomorph
:foo(<a b c>)              # value is a List
:foo([<a b c>])            # value is an Array
:foo({:42foo, :100bar})    # value is a Hash
:foo({.contains: 'meows'}) # the value is a Callable

In these form, you can simply omit the outer parentheses and let the inner brackets and curlies do their job:

:foo<42>                   # value is an IntStr allomorph
:foo<a b c>                # value is a List
:foo[<a b c>]              # value is an Array
:foo{:42foo, :100bar}      # value is a Hash
:foo{.contains: 'meows'}   # the value is a Callable

It looks a lot cleaner and is simpler to write. Both the Hash and Callable use the same set of curlies and the same simple rules as used by the {…} construct elsewhere in the language: if the content is empty, or contains a single list that starts with a Pair literal or %-sigiled variable, and the $_ variable or placeholder parameters are not used, a Hash is created; otherwise a Block (Callable) is created.

The angle bracket form (:foo<…>) follows the same rules as the angle bracket quoter used elsewhere in the language:

:foo< 42  >.value.^name.say; # OUTPUT: «IntStr␤»
:foo<meows>.value.^name.say; # OUTPUT: «Str␤»
:foo<a b c>.value.^name.say; # OUTPUT: «List␤»

And keep in mind that these two forms are not equivalent:

:42foo
:foo<42>

The first creates an Int object, while the second one creates an IntStr object, which is an allomorph. This difference is important for things that care about object identity, such as set operators

Sigiled Shortcut

The one thing I find a pain in the bit to write in other languages is constructs like this:

my $the-thing-with-a-thing = …
…
some-sub the-thing-with-a-thing => $the-thing-with-a-thing;

It's fairly common to name your variables the same as some named argument to which you wish to pass that variable as a value. The Perl 6 programming language offers a colonpair shortcut precisely for that case. Simply prepend a colon to the variable name to construct a colonpair with the key named the same as the variable (without including the sigil) and the value being the value of that variable. The only catch is the variable must have a sigil, so you can't use this shortcut with sigilless variables or constants.

my $the-thing-with-a-thing = …
…
some-sub :$the-thing-with-a-thing;

You'll notice that the syntax above looks exactly like how you'd declare a parameter that takes such a named argument—consistency is a good thing. All available sigils and twigils are supported, which makes the full list of variants for this shortcut look something like this:

# Name and value from variable
:$foo;  # same as :foo($foo)
:$*foo; # same as :foo($*foo)
:$?foo; # same as :foo($?foo)
:$.foo; # same as :foo($.foo)
:$!foo; # same as :foo($!foo)
:@foo;  # same as :foo(@foo)
:@*foo; # same as :foo(@*foo)
:@?foo; # same as :foo(@?foo)
:@.foo; # same as :foo(@.foo)
:@!foo; # same as :foo(@!foo)
:%foo;  # same as :foo(%foo)
:%*foo; # same as :foo(%*foo)
:%?foo; # same as :foo(%?foo)
:%.foo; # same as :foo(%.foo)
:%!foo; # same as :foo(%!foo)
:&foo;  # same as :foo(&foo)
:&*foo; # same as :foo(&*foo)
:&?foo; # same as :foo(&?foo)
:&.foo; # same as :foo(&.foo)
:&!foo; # same as :foo(&!foo)

This about wraps up the list of currently available colonpair shortcuts. As you can see, the huge list of shortcuts was reduced to a few simple patterns to follow. However, this might not be all the shortcuts that will exist for all the time…

The Future!

While currently aren't available, the following two shortcuts might become part of the language in future language versions.

The first one is the indirect lookup shortcut. If you have a named variable and the name of that variable in another variable, you can access the value of the first variable using the indirect lookup construct:

my $foo = "bar";
my %bar = :42foo, :70bar;
say %::($foo); # OUTPUT: «{bar => 70, foo => 42}␤»

If you squint, the indirect lookup is sort'f like a sigilled variable and colonpair shortcuts for sigilled variables exist, so it makes sense for the language to be consistent and support indirect lookup colonpair shortcut, which would look something like this, where the value of $foo contains the name of the key for the colonpair.

:%::($foo)

This form is currently listed as simply unimplemented feature in R#1532, so it'll likely see life some day.

The second possible future construct is the :.foo form, which was proposed in RFC R#1462. This form calls method .foo on the $_ topical variable and uses the return value as the value for the created Pair, with the name of the method being the name of the key.

This form comes up semi-frequently when you're passing values of attributes of one object to another with similarly-named attributes, so something like this:

Some::Other.new: :foo(.foo) :bar(.bar) :ber(.ber) with $obj

Would in shortcut form be written like this:

Some::Other.new: :.foo :.bar :.ber with $obj

At the time of this writing, this RFC has been self-rejected, but you never know if there'd be more calls for introduction of this syntax.

PART II: Use

Now that we're familiar with how to write all the forms of colonpairs, let's take a look at some of their available uses, especially those with special rules.

Parameters

To specify that a parameter should be a named rather than a positional parameter, simply use the sigilled variable colonpair shortcut:

sub meow($foo, :$bar) {
    say "$foo is positional and $bar is named";
}
meow 42, :100bar; # 42 is positional and 100 is named
meow :100bar, 42; # 42 is positional and 100 is named

Since parameters need some sort of a variable to bind their stuff to, pretty much all other forms of colonpairs are not available for use in parameters. This means that you can't, for example, declare sigilless named parameters and must instead explicitly use the is raw trait to get the rawness:

sub meow (\foo, :$bar is raw) {
    (foo, $bar) = ($bar, foo)
}
my $foo = 42;
my $bar = 100;
meow $foo, :$bar;
say [$foo, $bar]; # OUTPUT: «[100 42]»

The one other colonpair form available in parameters is the standard form that is used for aliasing multiple named params to the same name and parameter descructuring:

sub meow (:st(:nd(:rd(:$nth))), Positional :list(($, $second, |))) {
    say [$nth, $second];
}
meow :3rd, :list<a b c>; # OUTPUT: «[3 b]»

Pro-tip: if you're using the Rakudo compiler you may wish to take it easy with aliasing. Using aliases more than 1 level deep will cause the compiler to switch to the slow-path binder, which, as the name suggests, is about 10x slower.

A trick you can use is to use more than one parameter, each with aliases at most 1 level deep, and then merge them in the body:

sub meow (:st(:$nd is copy), :rd(:$nth)) {
    $nd //= $nth;
}

Argument Lists

Use of colonpairs in argument lists deserves a separate section due to a rule that's subtle enough to earn a spot in language's traps section. The rule involves the problem that a programmer may wish to pass Pair objects in argument lists as either a named or a positional argument.

In majority of cases, the colonpairs will be passed as named arguments:

sub args {
    say "Positional args are: @_.perl()";
    say "Named      args are: %_.perl()";
}

args :foo, :50bar, e => 42;
# OUTPUT:
# Positional args are: []
# Named      args are: {:bar(50), :e(42), :foo}

To pass a Pair object as a positional argument, you can do any of the following:

1. Wrap the entire colonpair in parentheses
2. Call some method on the colonpair, such as .self or .Pair; weird stuff like using R meta op on the => operator applies as well
3. Quote the key in foo => bar syntax
4. In foo => bar syntax, use a key that is not a valid identifier
5. Put your Pairs in a list and slip it in with the | "operator"

Here's that list of options in code form:

my @pairs := :42foo, :70meow;
args :foo.Pair, (:50bar), "baz" => "ber", e R=> 42, 42 => 42, |@pairs;

# OUTPUT:
# Positional args are: [:foo, :bar(50), :baz("ber"),
#   42 => 2.718281828459045e0, 42 => 42, :foo(42), :meow(70)]
# Named      args are: {}

Number (3) is especially worth keeping in mind if you're coming from other languages, like Perl 5, that use the fat arrow (=>) for key/value separation. This construct gets passed as a named argument only if the key is unquoted and only if it's a valid identifier.

Should it happen that you have to use one of these constructs, yet wish to pass them as named arguments instead of positionals, simply wrap them in parentheses and use the | prefix to "slip" them in. For the list of Pairs we were already slipping in in previous example, you'll need to coerce it into a Capture object first, as Pairs stuffed into a Capture—unlike a list—end up being named parameters, when the Capture is slipped into the argument list:

my @pairs := :42foo, :70meow;
args |(:foo.Pair), |(:50bar),   |("baz" => "ber"),
     |(e R=> 42),  |(42 => 42), |@pairs.Capture;

# OUTPUT:
# Positional args are: []
# Named      args are: {"42" => 42, :bar(50),
#                      :baz("ber"), :foo(42), :meow(70)}

The same slipping trick can be used to provide named arguments conditionally:

sub foo (:$bar = 'the default') { say $bar }

my $bar;
foo |(:bar($_) with $bar);  # OUTPUT: «the default␤»
$bar = 42;
foo |(:bar($_) with $bar);  # OUTPUT: «42␤»

If $bar is not defined, the with statement modifier will return Empty, which when slipped with | will end up being empty, allowing the parameter to attain its default value. Since |(:) looks like a sideways ninja, I call this technique "ninja-ing the arg".

Auto-Quoting in => Form

The sharp-eyed in the audience might have noticed the e => 42 colonpair in the previous section used letter e as a key, yet in reversed form e R=> 42, the e became 2.718281828459045e0, because the core language has e defined as Euler's number.

The reason it remained a plain string e in the e => 42 form is because this construct auto-quotes keys that are valid identifiers and so they will always end up as strings, even if a constant, a sigilless variable, or a routine with the same name exists:

my \meows = '🐱';
sub ehh { rand }
say %(
    meows => 'moo',
    ehh   => 42,
    τ     => 'meow',
); # OUTPUT: «{ehh => 42, meows => moo, τ => meow}␤»

A multitude of ways exist to avoid this autoquoting behaviour, but I'll show you just one that's good enough: slap a pair of parentheses around the key:

my \meows = '🐱';
sub ehh { rand }
say %(
    (meows) => 'moo',
    (ehh)   => 42,
    (τ)     => 'meow',
);
# OUTPUT: «{0.58437052771857 => 42, 6.283185307179586 => meow,
#           🐱 => moo}␤»

Simple!

Conclusion

That's pretty much all there is to know about colonpairs. We learned they can be used to construct Pair objects, used as adverbs, and used to specify named arguments and parameters.

We learned about various shortcuts, such as using key only for boolean True, sticking the negation operator or an unsigned integer between the colon and the key to specify boolean False or an Int value. We also learned that parentheses can be omited on colonpair values if there's already a set of curly, square, or angle brackets around the value and that prepending a colon to a sigilled variable name will create a colonpair that will use the name of that variable as the key.

In the second half of the article, we went over available colonpair syntaxes when specifying named paramaters, the pecularities in passing colonpairs as either named or positional arguments, as well as how to avoid auto-quoting of the key by wrapping it in parentheses.

I hope you found this informative.

-Ofun

This document is the June, 2018 progress report for The Perl Foundation's Perl 6 Constant and Rationals Grant.

Tangibles

The bonus deliverable "Perl 6 Numerics" Language documentation page was merged to master. It describes all of the available Perl 6 numerics, their interactions, suitability, and hierarchy.

The bulk of work on constants also has been merged to post-release-2018.06 branch, which will be merged to master after this month's release. I wrote 200 spec tests, available in S04-declarations/constant-6.d.t spec file, and about 500 words of documentation to cover this work.

Rationals

Since my last report, I first continued working on Rationals, focusing on three pieces of work that currently reside in car-grant-unreduce branch

1. Fixing the rare data race and doing some optimizations
2. Fixing bad math in some ops with Zero-Denominator Rationals (ZDRs)
3. Attempting a trial implementation where ZDRs are marked with a role, allowing us to improve performance of some operators.

The (1) was successful and I already was able to optimized some ops due to Rationals being always reduced now: == was made 28% faster and === was made 52% faster. I also made creation of Rationals 19% faster and argless Rational.round 4.7x faster (used by .Str and .base).

The plan for (2) was to try normalization to <1/0>, <0/0>, <-1/0> and the hope was that alone would fix all math problems. However, after that change some issues still remained. Also, doing this normalization created a new problem where code like say 42 / 0 would throw an Exception with message "division of 1 by 0 attempted" which is quite confusing for users who did not internalize that / op with Int objects is really just a Rat constructor. The 6.c spec actually covers this exact scenario and expects the thrown Exception to report value 42 for numerator, thus blocking this change.

The first attempt at (3) ended with a dead-end. Marking ZDRs with a role created an extra dispatch ambiguity with some operators like cmp. We already had to disambiguate that op with is default marker to disambiguate between Rational and Real candidates, so now we'd need an is default of defaults trait :). I gave up on this for now, but will likely revisit and try again.

As you can see, the Rationals work was a mixed bag, so to experiment with the problem space a bit I decided to implement my own rationals outside of core from scratch. The work is available in zoffixznet/perl6-Rashnl repo, which might become an ecosystem module if it offers significantly better rationals.

I doubt the entirety of Rashnl could be made core, as my current approach does not involve separate roles at all and just has a single class with a flag for fattiness that marks what in core is a FatRat type. The primary purpose of that work is to experiment with code and learn some lessons that could be applicable for core Rationals and achieve the goals of this Grant.

Due to that detour for the Rationals work and more time to think required, I switched to the work on constants for the time being…

Constants

The bulk of the work on constants is now completed and has been merged to post-release-2018.06 branch, which will be merged to master after this month's release.

I wrote 200 spec tests, available in S04-declarations/constant-6.d.t spec file, about 500 words of documentation, and 8 commits of the implementation.

Except for native types, the type constraints are now enforced on constants. The auto-coercive behaviour for %- sigilled constants was blocked by one test in 6.c specification and so that behavior has been added to 6.d language and currently requires the use of use v6.d.PREVIEW pragma to enable (6.c behavior is to simply throw without any attempts to coerce, making constant %foo = :42foo, :70bar fail, because it's a List).

One of the remaining things for constants is improving error reporting for unsupported constructs. Most likely I will implement support for coercers—even though they're currently not available on variables, there should be no problem with executing them during constant creation.

The other remaining item is natively-typed constants. Currently, my int const foo = 42 actually does not create a natively-typed constant at all. I hope the knowledge I'll gain while implementing the Grant's bonus work for support of native-typed unsigned attributes will help me in implementing the natively-typed constants as well.

No Report for July / Further Work

As has been discussed with and approved by my grant manager, I plan to now take a month off working on this grant, so there will be no report in mid-July, and the next report will be in mid-August. I plan to take this time to work on resolving Rakudo's outstanding spectest issues on Windows and possibly resolving some of the open Issues with our community websites.

After that period, I will resume working on the grant, finishing the remaining work on constants, and then going back to Rationals, and finally finishing the bonus work with the natively-typed entities.

Read this article on Rakudo.Party

If you follow the updates on KickStarter, you may know the Learning Perl 6 book is nearing completion, with the author planning to submit final manuscript to O'Reilly on June 18th.

What this means is the July's Rakudo Star release will possibly be the release the first crop of readers of that book will be using (the next release after that is in October). I've seen several people say they're waiting for this book to get published before they give Perl 6 a try. Coupled with the marketing the author and O'Reilly will be doing for the book, I expect to see an influx of new users.

For that reason, I'm making a call to action, for everyone to polish the experience of the first steps those users will make in Perl 6.

How to Help?

There are several things you can help with, depending on your skillset. And before anyone protests, don't worry, there's one thing everyone is able to do…

Install Perl 6

The easiest thing you can do is grab a clean box and try to install Perl 6 to it. This is especially useful if you've never done it before and have no idea what you're doing, as it'll be very easy for you to see things that need to be improved.

Are you having trouble finding or choosing what to install? Getting installation errors? Having trouble finding support channels? Report it!

Unless you can think of a better place, you can always report these things in our User Experience repo by filing a new Issue.

Learn/Use Perl 6

This, again, is the area where the less experience with Perl 6 you have, the better. Is there something you find difficult to use or hard to find? Let us know.

Code editors, books, documentation, modules, tutorials, excercises, contests, language news, support channels. If you were looking for any of those things and had a hard time finding them, the process will likely need to be addressed.

Rakudo on Windows

This is the area where I hope we'll make a lot of progress before the next Rakudo Star release (it'll be in July, based on Rakudo compiler release scheduled for July 21st). Few, if any of the core devs use Windows as their development environment, so the state of Rakudo on Windows is slightly lagging behind.

While 6.d-proposals roast stresstest is clean on Linux and MacOS, on Windows, there's a bunch of test failures. Several of them are likely problematic tests themselves (e.g. those that shell out and expect cmd.exe to handle Unicode out of the box). When I last looked at the failing tests, some of them were failing due to how &run escapes arguments; since perl6 is launched with a batch file on windows, using $*EXECUTABLE with &run would require using cmd.exe-style escapes of command line arguments, which &run doesn't use. There's some discussion for this issue on R#1325, RT#132258, and self-rejected RFC R#1306.

If you'd like to look into these problems, you can install Rakudo from source on Windows and then run gmake stresstest (or whatever make equivalent you have) to clone all the spectests into t/spec directory and run them, so you'll be able to see what's failing. You can run individual tests with t/fudgeandrun t/spec/42-foobar/the-test-file.t

There's also a second item of lesser importance: Rakudo Star build on 32-bit Windows. The latest build we have for that system is 2016.01, which is quite outdated. On occasion people do ask for 32-bit builds and currently we can only suggests to build from source.

It'd be nice to have a more recent build created. I'm unfamiliar with what's involved. If you're interested in helping. Join our IRC chat and try to speak to stmuk or FROGGS

Help Resolve Issues

Help us resolve open Issues. In the context of this call to action, the most pertinent repos would likely be User Experience, Perl6.org website, Modules.Perl6.org website, as well as Docs, Rakudo Star, Rakudo itself, and Roast Test Suite.

For core hacking resources, there are some tutorials with "Core Hacking" in their titles on Rakudo.Party website and the NQP/Rakudo Internals Course and 6guts blog. I also like to use Z-Script rakudo dev helper script.

Conclusion

The LP6 book is about to hit the shelves and will likely bring a crop of new Perl 6 users. We can improve the perception of the language by polishing the experience of those users. Let's see if there are any problems with obtaining the compiler or any resources a begginer would need to get started with the language.

There are also some issues on Windows that need to be addressed, such as failing stresstests and 32-bit Rakudo Star builds.

If you can give a hand with any of that, it'd be greatly appreciated. File a new Issue in our User Experience repo or just chat with us on IRC.

-Ofun

This document is the May, 2018 progress report for The Perl Foundation's Perl 6 Constant and Rationals Grant.

Tangibles Produced

This month I was working in docs and roast repos, in separate car-grant-midrat branches. So far, they contain 29 documentation commits and 11 spec commits adding about 5,824 words of documentation and 152 tests.

The tests specced the MidRat type. The documentation, along with documenting MidRat/MidRatStr types, centered largely around new Language/Numerics page that describes all of the available Perl 6 numeric types—including native and atomic numerics—their purpose, properties, and hierarchy. This guide was not on the list of deliverables of the Grant and has been produced as a bonus item.

General Info

I started the grant by working on adding the proposed MidRat/MidRatStr type pair. This was the most contentious part of the Rationals Work Proposal that before the grant underwent three revisions, settling on solving the problem by adding the MidRat type.

I went full-on with Test/Documentation Driven Development and I think the result is a fantastic example of the power of this approach. After adding 5,824 words of docs and 152 tests, I still haven't written a single line of code of the implementation in the compiler itself.

The process highlighted several issues with MidRat type and I've tentatively decided not to implement it at this time. Its functionality will be subsumed with the plain Rat type. The plan to change Rat type to use uint64 native type for the denominator is tentatively canceled and instead an Int denominator will accept denominators over 64 bits in size in all the cases where a MidRat type were going to be produced. At the same time, the Rat will degrade to a Num in all the cases where a MidRat were to do so.

Doing so robs us of potential performance improvements due to using native denominator, however we still have the option of implementing a MidRat-type solution open in the future. Were we to go with implementing MidRat right now, we'd be locking ourselves into supporting it basically forever, and exposing ourselves to yet unforeseen design issues.

Issues/Clarifications of MidRat

What is a .Numeric/.Real of a MidRat?

Writing documentation first showed an ambiguity with what .Numeric/.Real methods should return for a MidRat. The work proposal had MidRat as an allomorph of Rat and FatRat, and while .Numeric on IntStr allomorph returns its numeric component, a MidRat would have two such components.

This issue is complicated by the fact that prefix:<+> uses .Numeric, so that means 0 + MidRat.new(1,2) would produce a Rat objet, yet +MidRat.new(1,2) would produce a potentially different object. To maintain consistency there, at the time I settled on simply having .Numeric/.Real coerce the MidRat to a Rat, which means that had the potential of degradation of that Rat to a Num. Bring the MidRatStr into play and you get three different types by chaining .Numeric calls. That was already slightly weird.

Convergence to Plain Rat

While documenting infectiousness of numerics, the question of infectiousness of MidRat came about. The original work proposal was vague in this area but after documenting the MidRat type, it became obvious that its infectiousness must be the same as if Rat type was used instead. That's the first point of convergence to Rat.

The second became apparent when writing degradation tests. The work proposal argued for MidRat to be an allomorph of Rat and FatRat types. A MidRat has infectiousness of a Rat. But that means this code…

my FatRat $a = get-rational;
my FatRat $b = get-rational;
say $a + $b;

…has the potential to produce a Num of all things (MidRats degrade to Rat and that can degrade to a Num). That's certainly unacceptable, so by this point, I decided to toss FatRat from the ancestors of MidRat, leaving it be just a subclass of Rat. This largely resolved the issue of what to return from .Numeric/.Real methods.

...

By now you can see that the MidRat got reduced to just being a Rat with non-native denominator. Adding two extra types (MidRat/MidRatStr) to an already populous zoo of Perl 6 numerics for such a tiny detail was hard for me to justify. So, after flipping a coin to confirm my gut feeling, I decided to abandon the idea of MidRat and to repurpose the Rat type to resolve all of the issues MidRat were meant to resolve.

The documentation and tests I wrote won't go to waste: they're simply going to be modified to document/test the MidRat's functionally from the point of view of the Rat type that subsumed it.

Going Forward

I still plan on doing the Bonus Work of the grant, fixing bugs in native uint64 attributes, despite no longer planning to use them directly in the denominator of the Rat type. I'll also try to make a perf improvement by selecting native/non-native NQP ops for dividing Rats, to make up for not using native attributes in them.

In lieu of implementing the MidRat type originally promised, I'll work on resolving some additional bugs. Likely trying to resolve value drift in MoarVM Numification of Rationals that currently does not select the closest representable Num.

For the next month, I plan to work more on Rationals, fixing the data race and normalization of zero-denominator Rationals.

Related Work Done By Others

• AlexDaniel++ fixed object-identity bug in Rationals. The fix was done using the .REDUCE-ME method, which has the data race and will go away in about a month, but in the meantime, object identity will be correct.
• thundergnat++ did some work on improving performance of Rat.Str and FatRat.Str and fixed an issue with unwanted trailing zeros.

Read this article on Rakudo.Party

The Perl 6 programming language had a turbulent birth. It was announced in the summer of 2000 and the first stable language release shipped out only 2 years ago, on Christmas, 2015. A lot has happened during that decade and a half, yet the details are hard to piece together.

After my recent facelift to rakudo.org, I'm working on a (second) facelift to perl6.org website.

Part of the work involves bringing all the Perl 6 deliverables under one umbrella, so the user isn't thrown around multiple websites, trying to find what to install. At the same time, we want to strengthen the distinction between Perl 6 the language and the compilers that implement it, as well as encourage more implementors to give it a go at implementing a Perl 6 programming language compiler.

The Perl 6 Programming Language Museum will be part of that effort and along with interesting tidbits of Perl 6 history, it'll showcase past implementation attempts that may no longer be in active development today. Since I don't know much about what happened before I came to the language sometime in 2015, I need your help in collecting those tidbits.

Larry Wall at FOSDEM 2015, photo by Klapi

In my mind's eye, I'm imagining a few pages on perl6.org; something in the same vein as Computer History Museum's pages—pictures, years, and info, and potentially links to code repositories. Depending on the content we collect, it's possible there will be a digital PDF version of the Museum that can also be printed and handed out at events, if desired.

I'm looking for:

• Descriptions of interesting/significant events (like the mug throwing incident).
• Descriptions of interesting/significant implementations of Perl 6 or influential Perl 6 projects. Having links to repos/tarballs of their code is a plus.
• Samples of interesting/significant email threads or chat logs.
• Pictures of interesting/significant objects (first sight at plush Camelias?).
• Pictures of interesting/significant humans (a filled out model release form is required).
• Anything else that's Museum worthy.

If you have any of these items, please submit them to the appropriate year directory in the Perl 6 Museum Items repository. If you're a member of Perl 6 GitHub org, you should already have a commit bit to that repo. Otherwise, submit your items via a pull request.

Let's build something cool and interesting for the people using Perl 6 a hundred years from now to look at and remember!

If you have any questions or need help, talk to a human on our IRC chat.

-OFun

Read this article on Rakudo.Party

Recently, I came across a somewhat-frantic comment on StackOverflow that describes a 2017.01 change to the type of return value of .sort:

"you just can't be sure what ~~ returns" Ouch. […] .list the result of a sort is presumably an appropriate work around. But, still, ouch. I don't know of a blog post or whatever that explains how P6 approaches changes to the language; and to roast; and to Rakudo. Perhaps someone will write one that also explains how this aspect of 2017.01 was conceived, considered and applied; what was right about the change; what was wrong; etc.

Today, I decided to answer that call to write a blog post and reply to all of the questions posed in the comment, as well as explain how it's possible that such an "ouch" change made it in.

On Versioning

The '6' in Perl 6 is just part of the name. The language version itself is encoded by a sequential letter, which is also the starting letter of a codename for that release. For example, the current stable language version is 6.c "Christmas". The next language release will be 6.d with one of the proposed codenames being "Diwali". The version after that will be 6.e, then 6.f, and so on.

If you've used Perl 6 sometime between 2015 and 2018, you likely used the "Rakudo" compiler, which is often packaged as "Rakudo Star" distribution and is versioned with the year and the month of the release, e.g. release 2017.01.

In some languages, like Perl 6's sister language Perl 5, what the compiler does is what the language itself is. Bugs aside, if the latest (2017.09) Perl 5 compiler gives 4 for 2+2, then that's the definition of what 2+2 is in the Perl 5 language.

In Perl 6, however, how a compiler (e.g. "Rakudo") behaves or what it implements does not define the Perl 6 language. The Perl 6 language specification does. The specification consists of a test suite of about 155,000 tests and anything that passes that test suite can call itself a "Perl 6 compiler".

It's to this specification version 6.c "Christmas" refers. It was released on December 25, 2015 and at the time of this writing, it's the first and only release of a stable language spec. Aside from a few error corrections, there were no changes to that specification… The latest version of Rakudo still passes every single test—it's a release requirement.

On Changes

Ardent Perl 6 users would likely recall that there have been many changes in the Rakudo compiler since Christmas 2015. Including the "ouch" change referenced by that StackOverflow comment. If the specification did not change and core devs are not allowed to make changes that break 6.c specification, how is it possible that the return type of .sort could have changed?

The reason is—and I hope the other core devs will forgive me for my choice of imagery—the specification is full of holes!

It doesn't (yet) cover every imaginable use and combination of features. What happens when you try to print a Junction of strings? As far as 6.c version of Perl 6 language is concerned, that's undefined behaviour. What object do you get if you call .Numeric on an Rat type object rather than an instance? Undefined behaviour. What about the return value of .sort? You'll get sorted values in an Iterable type, but whether that type is a Seq or a List is not specified by the 6.c specification.

This is how 2017.01 version of Rakudo managed to change the return type of .sort, despite being a compliant implementation of the 6.c language—the spec was not precise about what Iterable type .sort must return; both Seq and List are Iterable, thus both conform to the spec. (It's worth noting that since 2017.01 we implemented an extended testing framework that also guides our decisions on whether we actually allow changes that don't violate the spec).

In my personal opinion, the 6.c spec is overly sparse in places, which is why we saw a number of large changes in 2016 and early 2017, including the "ouch" change the commenter on StackOverlow referred to. But… it won't stay that way forever.

The Future of the Spec

At the time of this writing, there have been 3,129 commits to the spec, since 6.c language release. These are the proposals for the 6.d language specification. While some of these commits address new features, a lot of them close those holes the 6.c spec contains. The main goal is not to write a "whole new spec" but to refine and clarify the previous version.

Thus, when 6.d is released, it'll look something like this:

A few more slices of new features, but largely the same thing. Still some holes (undefined behaviour) in it, but a lot less than in 6.c language. It now defines that printing a Junction will thread it; that calling .Numeric on a Numeric type object gives a numeric equivalent of zero of that type and a warning; and that the .sort's Iterable return type is a Seq, not a List.

As more uses of combinations original designers haven't thought of come around, even more holes will be covered in future language versions.

Breaking Things

The cheese metaphor covers refinements to the specification, but there's another set of changes the core developers sometimes have to make: changes that violate previous versions of the specification. For 6.d language, the list of such changes is available in our 6.d-prep repository (some of the listed changes don't violate 6.c spec, but still have significant impact so we pushed them to the next language version).

This may seem to be a contradiction: didn't I say earlier that passing 6.c specification is part of the compiler's release requirements? The key to resolving that contradiction lies in ability to request different language versions in different comp units (e.g. in different modules) that are used by the same program.

A single compiler can support multiple language versions. Specifying use v6.c pragma loads 6.c language. Specifying use v6.d (currently available as use v6.d.PREVIEW) loads 6.d language. Not specifying anything loads the newest version the compiler supports.

One of the changes between 6.c and 6.d languages is that await no longer blocks the thread in 6.d. We can observe this change using a single small script that loads two modules. The code between the two modules is the same, except they request different language versions:

# file ./C.pm6
use v6.c;
sub await-c is export {
    await ^10 .map: {
        start await ^5 .map: { start await Promise.in: 1 }
    }
    say "6.c version took $(now - ENTER now) secs";
}

# file ./D.pm6
use v6.d.PREVIEW;
sub await-d is export {
    await ^10 .map: {
        start await ^5 .map: { start await Promise.in: 1 }
    }
    say "6.d version took $(now - ENTER now) secs";
}

# $ perl6 -I. -MC -MD -e 'await-c; await-d'
# 6.c version took 2.05268528 secs
# 6.d version took 1.038609 secs

When we run the program, we see that no-longer blocked threads let 6.d version complete a lot faster (in fact, if you bump the loop numbers by a factor, 6.d would still complete, while 6.c would deadlock).

So this is the Perl 6 mechanism that lets the core developers make breaking changes without breaking user's programs. There are some limitations to it (e.g. methods on classes)—so for some things there still will be standard deprecation procedures. We also try to limit the number of such spec-breaking changes, to reduce the maintenance burden and impact on users who don't want to lock their code down to some older version. Thus, don't worry about getting some weird new language on the next language release—the differences will be minimal.

Who Decides?

This all brings us to one of the questions posed by that StackOverflow user: how do language changes get conceived, considered, and applied—in short: who decides what the behaviour is to be like? What is the process?

As far as conception goes, many of the current ideas are based on seeing what our users need. Some proposals come directly from users; others get inspired as more elegant solutions to problems users showed they were trying to solve. Some of the changes proposed for 6.d language were informed by problematic areas of currently-implemented features that weren't foreseen during original implementation.

When it comes to implementation, the scope of the feature and core developer's expertise with the given area of the codebase generally drive the process. With the "ouch" change, the expert in the area of Iterables deemed Seq to be a superior type for .sort to return, due to its non-caching behaviour as well as its ease of degenerating into a caching List.

Some changes get opened as an Issue on the bugtracker first, to notify other devs of the impending change. Large changes usually get a proposed design written down first. The proposal is shared with the core devs and feedback is gathered before the proposal is actually implemented. The implementation of significant things is also merged far away from the date of the next release, to let the bleeding-edge users find any potential problems in the work.

Geth, our IRC bot, announces all commits in our development IRC channel. Most of the core devs backlog that channel, so any of the potentially problematic commits—even if one of the devs goes ahead and commits the change—get discussed and at times reverted.

The Perl 6 pumpking (Jonathan Worthington) and the BDFL (Larry Wall) are available to provide feedback on controversial, questionable, or large changes being proposed. They also have the veto power on any changes. Our messaging bot helps us request feedback from them, even if they're currently not in the chat.

When it comes to errata to previous specifications, unless the test to be changed is "obviously wrong", the decision on whether the errata can be applied is delegated to the Release Manager (AlexDaniel), and informed by the pumpking/BDFL, if required.

The Future

The current process is a bit loose in places. A test that's "obviously wrong" to one person might have some valid reasons behind it to someone else. This is why the TODO for 6.d release lists several documents to be written that will refine the procedures for various types of changes.

It won't be on the scale of PEP, but simply something more concrete for the core devs to refer to, when performing changes that have some impact on the users. It's a balancing act between organization and procedure and letting through a consistent flow of contributions.

And if breaking changes have to be made, an alert will be pushed to the the P6lert service for users of Perl 6 to get informed of them in advance.

Conclusion

Today, we gleaned an insight into how Perl 6 core devs introduce changes to the compiler and the language.

The language specification and the compiler's behaviour are separate entities. The 6.c language specification has places of unspecified behaviour, which is how changes that have large impact on the users slipped through in the past.

The extended testing framework as well as specification clarifications offered by 6.d language proposal tests that refine the specification and close the holes with undefined behaviour reduce unforeseen impact on the users.

The core dev team informs their decisions based on user's feedback and the way the language is used by the community. Large changes get written up as proposals and the pumking/BDFL offer advise on anything controversial.

In the future, more refined practices for how changes are made will be defined, as we work on making upgrade experience more predictable and non-breaking for our users. The P6lert service helps that goal and is already available today.

Hope this answers all the questions :)

Read this article on Rakudo.Party

Over the past month, I spent some time in Rakudo's QAST land writing a few optimizations, fixing bugs involving warnings, as well as squashing a monster hive of 10 thunk scoping bugs with a single commit. In today's article, we'll go over that last feat in detail, as well as learn what QAST is and how to work with it.

PART I: The QAST

"QAST" stands for "Q" Abstract Syntax Tree. The "Q" is there because it's comes after letter "P", and "P" used to be in "PAST" to stand for "Parrot", the name of an earlier, experimental Perl 6 implementation (or rather, its virtual machine). Let's see what QAST is all about!

Dumping QAST

Every Rakudo Perl 6 program compiles down to a tree of QAST nodes and you can dump that tree if you specify --target=ast or --target=optimize command line option to perl6 when compiling a program or a module:

$ perl6 --target=ast -e 'say "Hello, World!"'
[...]
- QAST::Op(call &say) <sunk> :statement_id<?> say \"Hello, World!\"
  - QAST::Want <wanted> Hello, World!
    - QAST::WVal(Str)
    - Ss
    - QAST::SVal(Hello, World!)
[...]

The difference between the --target=ast and --target=optimize is that the former shows the QAST tree as soon as it has been generated, while the later shows the QAST tree after the static optimizer has had a go at it.

While the command line option gives you the QAST for the entire program (excluding modules pre-compiled separately), each QAST::Node object has a .dump method you can use to dump specific QAST pieces of interest from within Rakudo's source code.

For example, to examine the QAST generated by the statement token, I'd find method statement in src/Perl6/Actions.nqp and stick nqp::say('statement QAST: ' ~ $past.dump) close to the end of the method.

Since Rakudo's compilation takes a couple of minutes for each go, I like to key my debug dumps on env variables, like this:

nqp::atkey(nqp::getenvhash(),'ZZ1') && nqp::say('ZZ1: something or other');
...
nqp::atkey(nqp::getenvhash(),'ZZ2') && nqp::say('ZZ2: something else');

Then, I can execute the compiled ./perl6 as if I didn't add anything, and enable my dumps by running ZZ1=1 ./perl6, ZZ2=1 ./perl6, or both dumps at the same time with ZZ1=1 ZZ2=1 ./perl6.

Viewing QAST

Looking at the output of --target dumps in the terminal is sufficient for a quickie glance at the trees, but for extra assistance you can install CoreHackers::Q module that brings in q command line utility.

Simply prefix your regular perl6 invocation with q a or q o to produce --target=ast and --target=optimize QAST dumps respectively. The program will generate out.html file in the current directory:

$ q a perl6 -e 'say "Hello, World!"'
$ firefox out.html

Pop open the generated HTML file and reap these benefits:

• Color-coded QAST nodes
• Color hints for sunk nodes
• Ctrl+Click on any node to collapse it
• Muted view of QAST::Want alternatives, makes it easier to ignore them

Eventually, I hope to extend this tool and make it more helpful, but at the time of this writing, that's all it does.

The QAST Forest

There are four main files in rakudo's source where you'd expect to be working with QAST nodes: src/Perl6/Grammar.nqp, src/Perl6/Actions.nqp, src/Perl6/World.nqp, and src/Perl6/Optimizer.nqp. If you're using Z-Script utility, you can even run z q command to open these four files in Atom editor.

Grammar.nqp is the Perl 6 grammar. Actions.nqp are the actions for it. World.nqp contains all sorts of helpful routines used by both Grammar.nqp and Actions.nqp that access them via the $*W dynamic variable containing a Perl6::World object. Lastly, Optimizer.nqp contains Rakudo's static optimizer.

The root (of all evil) is the QAST::Node object, with all the other QAST nodes being its subclasses. Let's review some of the popular ones:

QAST::Op

QAST::Op nodes are the workhorse of the QAST world. The :op named argument specifies the name of an NQP op or the name of a Rakudo's NQP extension op and its children are the arguments:

Here's a say op printing a string value:

QAST::Op.new: :op<say>,
  QAST::SVal.new: :value('Hello, World!');

And here's a QAST node for a call op that calls Perl 6's infix:<+> operator; notice how the name of the routine we call is given via :name named argument:

QAST::Op.new: :op<call>, :name('&infix:<+>'),
  QAST::IVal.new( :value(2)),
  QAST::IVal.new: :value(2)

QAST::*Val

The QAST::SVal, QAST::IVal, QAST::NVal, and QAST::WVal nodes, specify string, integer, float, and "World" object values respectively. The first three are the "unboxed" raw values, while World objects are everything else, such as DateTime, Block, or Str objects.

QAST::Want

Some of the objects can be represented by multiple QAST::*Val nodes, where the most appropriate value is used depending on what is wanted in the current context. QAST::Want node contains these alternatives, interleaved with string markers indicating what those alternatives are.

For example, numeric value 42 in Perl 6 could be wanted as an object to call some method on, or as a raw value to be assigned to a native int variable. The QAST::Want node for it would look like this:

QAST::Want.new:
  QAST::WVal.new(:value($Int-obj))),
  'Ii',
  QAST::IVal.new: :value(42)

The $Int-obj above would contain an instance of Int type with value set to 42. The Ii marker indicates the following alternative is an integer value and we provide a QAST::IVal object containing it. The other possible markers are Nn (float), Ss (string), and v (void context) alternatives.

When these nodes are later converted to bytecode, the most appropriate value will be selected, with the first child being the "default" value, to be used when none of the available alternatives make the cut.

QAST::Var

These nodes are used for variables and parameters. The :name named argument specifies the name of the variable and :scope its scope:

QAST::Op.new: :op('bind'),
  QAST::Var.new(:name<$x>, :scope<lexical>, :decl<var>, :returns(int)),
  QAST::IVal.new: :value(0)

The :decl named arg is present when the node is used for the variable's declaration (when it's absent, we simply reference the variable) and its value dictates what sort of variable it is: var for variables and param for routine parameters. Several other :decl types, as well as optional arguments specifying additional configuration of the variable exist. You can find them discussed in the QAST documentation

QAST::Stmt / QAST::Stmts

These are statement grouping constructs. For example, here, the truthy branch of an nqp::if contains three nqp::say statements, all grouped inside QAST::Stmts:

QAST::Op.new: :op<if>,
  QAST::IVal.new(:value(42)),
  QAST::Stmts.new(
    QAST::Op.new( :op<say>, QAST::SVal.new: :value<foo>),
    QAST::Op.new( :op<say>, QAST::SVal.new: :value<bar>),
    QAST::Op.new: :op<say>, QAST::SVal.new: :value<ber>),
  QAST::Op.new: :op<say>, QAST::SVal.new: :value<meow>,

The singular QAST::Stmt is similar. The difference is it marks a register allocation boundary, beyond which, any temporaries are free to be reused. When used correctly, this alternative can result in better code generation.

QAST::Block

This node is both a unit of invocation and a unit of lexical scoping. For example, code sub foo { say "hello" } might compile to a QAST::Block like this:

Block (:cuid(1)) <wanted> :IN_DECL<sub> { say \"hello\" }
[...]
  Stmts <wanted> say \"hello\"
    Stmt <wanted final> say \"hello\"
      Want <wanted>
        Op (call &say) <wanted> :statement_id<?> say \"hello\"
          Want <wanted> hello
            WVal (Str)
            - Ss
            SVal (hello)
        - v
        Op (p6sink)
          Op (call &say) <wanted> :statement_id<?> say \"hello\"
            Want <wanted> hello
              WVal (Str)
              - Ss
              SVal (hello)
[...]

Each block demarcates a lexical scope boundary—this detail comes into play in Part II of this article, when we'll be going over a fix for a bug.

Others

A few more QAST nodes exist. They're out of scope of this article, but you may wish to read the documentation or, since some of them are not appear in those docs, go straight to the source.

Executing QAST Trees

Having a decent familarity with nqp ops (as well as Rakudo's nqp extensions) is helpful when working with QAST. A sharp eye would notice in QAST dumps that many QAST::Op nodes correspond to nqp::* op calls, where :op named argument specifies the name of the op.

When writing large QAST trees, it's handy to write them down using pure NQP ops first, and then translate the result into a tree of QAST node objects. Let's look at a simplified example:

nqp::if(
  nqp::isgt_n(nqp::rand_n(1e0), .5e0),
  nqp::say('Glass half full'),
  nqp::say('Glass half empty'));

We have NQP op, so we'll start with QAST::Op node, using 'if' as the value for :op. The op takes three positional arguments—the three ops used for the conditional, the truthy branch, and the falsy branch. Some of the ops also take float and string values, so we'll use QAST::NVal and QAST::SVal nodes for those. The result is:

QAST::Op.new(:op('if'),
  QAST::Op.new(:op('isgt_n'),
    QAST::Op.new(:op('rand_n'),
      QAST::NVal.new(:value(1e0))
    ),
    QAST::NVal.new(:value(.5e0))
  ),
  QAST::Op.new(:op('say'),
    QAST::SVal.new(:value('Glass half full'))
  ),
  QAST::Op.new(:op('say'),
    QAST::SVal.new(:value('Glass half empty'))
  )
)

I find it easier to track the tree's nesting by using parentheses only when necessary, preferring colon method call syntax whenever possible:

QAST::Op.new: :op<if>,
  QAST::Op.new(:op<isgt_n>,
    QAST::Op.new(:op<rand_n>,
      QAST::NVal.new: :value(1e0)),
    QAST::NVal.new: :value(.5e0)),
  QAST::Op.new(:op<say>,
    QAST::SVal.new: :value('Glass half full')),
  QAST::Op.new: :op<say>,
    QAST::SVal.new: :value('Glass half empty')

If a .new is followed by a colon, there aren't any more nodes on the same level. If .new is followed by an opening parentheses, there are more sister nodes yet to come.

Due to Rakudo's lengthy compilation, it can be handy to execute your QAST tree without having to stick it into src/Perl6/Actions.nqp or similar file first. To some extent, it's possible to do that with a regular Perl 6 program. We'll simply access Perl6::World object in $*W variable inside a BEGIN block, where it still exists, and call .compile_time_evaluate method, giving it an empty variable as the first positional (it expects a Match object for the tree) and our QAST tree as the second positional:

use QAST:from<NQP>;
BEGIN $*W.compile_time_evaluate: $,
    QAST::Op.new: :op<if>,
      QAST::Op.new(:op<isgt_n>,
        QAST::Op.new(:op<rand_n>,
          QAST::NVal.new: :value(1e0)),
        QAST::NVal.new: :value(.5e0)),
      QAST::Op.new(:op<say>,
        QAST::SVal.new: :value('Glass half full')),
      QAST::Op.new: :op<say>,
        QAST::SVal.new: :value('Glass half empty')

The one caveat with this method is we're using full-blown Perl 6 language, whereas in src/Perl6/Actions.nqp and related files, as .nqp extension suggests, we're using NQP language only. Keep an eye out for strange explosions; it's possible your QAST tree that explodes in Perl 6 will compile just fine in the land of pure NQP.

Annotating QAST Nodes

All QAST nodes support annotations that allow you to attach an arbitrary value to a node and then read that value elsewhere. To add an annotation, use .annotate method, which takes two positional arguments—a string containing name of the annotation and the value to attach to it—and returns that value. Recent versions of NQP also have .annotate_self method that works the same, except it returns the QAST node itself:

$qast.annotate_self('foo', 42).annotate: 'bar', 'meow';

Later, you can read that value using .ann method that takes the name of the annotation as the argument. If the annotation doesn't exist, NQPMu is returned instead:

note($qast.ann: 'foo'); # OUTPUT: «42␤»

You can also check for whether an annotation merely exists using .has_ann method that returns 1 (true) or 0 (false):

note($qast.has_ann: 'bar'); # OUTPUT: «1␤»

Or dump all of the annotations on the node (to prevent potential flood of output, most values will be dumped as simply a question mark):

note($qast.dump_annotations); # OUTPUT: « :bar<?> :foo<?>␤»);

Lasty, to clear all annotations on the node, simply call .clear_annotations method.

Mutating QAST Nodes

A handy thing to do with QAST node objects is to mutate them into something better. That's essentially all the static optimizer in src/Perl6/Optimizer.nqp does. Named arguments can be mutated by calling them as methods and providing a value. For example, $qast.op('callstatic') will change the value of :op from whatever it is to callstatic. Positional arguments can be altered by re-assignment to a positional index, as well as shift, push, unshift, pop operations performed either via method calls with those names or nqp:: ops. Some nodes also support nqp::elems calls on them, which is slightly faster than the generic pattern of +@($qast) that can be used on all nodes to find out the number of children a node contains.

As an exercise, let's write a small optimization: some operations, like $foo < $bar < $ber compile to nqp::chain ops. That is so even if we have only two children, e.g. $foo < $bar. In such cases, rewriting the op to be nqp::call has performance advantages: not only nqp::call on its own is a little bit faster than nqp::chain, the static optimizer knows how to do further optimizations on nqp::call ops.

Let's take a look at what both 2-child and 2+-child nqp::chain chains look like:

$ perl6 --target=ast -e '2 < 3 < 4; 2 < 3'

The first statement compiled to this (I removed QAST::Wants for clarity):

- QAST::Op(chain &infix:«<»)  :statement_id<?> <
  - QAST::Op(chain &infix:«<») <wanted> <
    - QAST::IVal(2)
    - QAST::IVal(3)
  - QAST::IVal(4)

And the second one to:

- QAST::Op(chain &infix:«<»)  :statement_id<?> <
  - QAST::IVal(2)
  - QAST::IVal(3)

Thus, to target our optimization correctly, we need to ensure neither child of our chain op is a chain op. In addition, we need to ensure that the op we're optimizing is not itself a child of another chain op.

Raking the code of the optimizer, we can spot that chain depth is already tracked via $!chain_depth attribute, so we merely need to ensure we're at the first link of the chain. The code then becomes:

$qast.op: 'call'
  if nqp::istype($qast, QAST::Op)
  && $qast.op eq 'chain'
  && $!chain_depth == 1
  && ! (nqp::istype($qast[0], QAST::Op) && $qast[0].op eq 'chain')
  && ! (nqp::istype($qast[1], QAST::Op) && $qast[1].op eq 'chain');

Once we find a chain QAST::Op, we index into it and use nqp::istype to check the type of kid nodes, and if those happen to be QAST::Op nodes, we ensure the :op parameter is not a chain op. If all of the conditions are met, we simply call .op method on our node with value 'call' to convert it into a call op.

We then stick our optimization early enough into .visit_op method of the optimizer and its later portions will further optimize our call.

A fairly easy and straightforward optimization that can bring a lot of benefit.

PART II: A Thunk in The Trunk

Note: it took me three evenings to debug and fix the following tickets. To learn the solution I tried many dead ends that I won't be covering, to keep you from getting bored, and instead will instantly jump to conclusions. The point I'm making is that fixing core bugs is a lot easier than may seem from reading this article—you just need to be willing to spend some time on them.

Now that we have some familiarity with QAST, let's try to fix a bug that existed in Rakudo v2018.01.30.ga.5.c.2398.cc and earlier. The ticket in question is R#1212, that shows the following problem:

$ perl6 -e 'say <a b c>[$_ xx 2] with 1'

Use of Nil in string context
  in block  at -e line 1
Unable to call postcircumfix [ (Any) ] with a type object
Indexing requires a defined object
  in block <unit> at -e line 1

It looks like the $_ topical variable inside the indexing brackets fails to get the value from with statement modifier and ends up being undefined. Sounds like a challenge!

It's A Hive!

Both with and xx operator create thunks (thunks are like blocks of code, without having explicit blocks in the code; this, for example, lets rand xx 10 to produce 10 different random values; rand is thunked and the thunk is called for each iteration). This reminded me of some other tickets I've seen, so I went to fail.rakudo.party and looked through open tickets for anything that mentioned thunking or wrong scoping.

I ended up with a list of 7 tickets, and with the help of dogbert++ later increased the number to 9, which with the original Issue gives us a total of 10 different manifestations of a bug. The other tickets are RT#130575, RT#132337, RT#131548, RT#132211, RT#126569, RT#128054, RT#126413, RT#126984, and RT#132172. Quite a bug hive!

Test It Out

Our starting point is to cover each manifestation of the bug with a test. Make all the test pass and you know you've fixed the bug, plus you already have something to place into roast, to cover the tickets. My tests ended up looking like this, where I've used gather/take duo to capture what the tickets' code printed to the screen:

use Test;
plan 1;
subtest 'thunking closure scoping' => {
    plan 10;

    # https://github.com/rakudo/rakudo/issues/1212
    is-deeply <a b c>[$_ xx 2], <b b>.Seq, 'xx inside `with`' with 1;

    # RT #130575
    is-deeply gather {
        sub itcavuc ($c) { try {take $c} andthen 42 };
        itcavuc $_ for 2, 4, 6;
    }, (2, 4, 6).Seq, 'try with block and andthen';

    # RT #132337
    is-deeply gather {
        sub foo ($str) { { take $str }() orelse Nil }
        foo "cc"; foo "dd";
    }, <cc dd>.Seq, 'block in a sub with orelse';

    # RT #131548
    is-deeply gather for ^7 {
        my $x = 1;
        1 andthen $x.take andthen $x = 2 andthen $x = 3 andthen $x = 4;
    }, 1 xx 7, 'loop + lexical variable plus chain of andthens';

    # RT #132211
    is-deeply gather for <a b c> { $^v.uc andthen $v.take orelse .say },
        <a b c>.Seq, 'loop + andthen + orelse';

    # RT #126569
    is-deeply gather { (.take xx 10) given 42 }, 42 xx 10,
        'parentheses + xx + given';

    # RT #128054
    is-deeply gather { take ("{$_}") for <aa bb> }, <aa bb>.Seq,
        'postfix for + take + block in a string';

    # RT #126413
    is-deeply gather { take (* + $_)(32) given 10 }, 42.Seq,
        'given + whatever code closure execution';

    # RT #126984
    is-deeply gather {
        sub foo($x) { (* ~ $x)($_).take given $x }; foo(1); foo(2)
    }, ("11", "22").Seq, 'sub + given + whatevercode closure execution';

    # RT #132172
    is-deeply gather { sub {
        my $ver =.lines.uc with "totally-not-there".IO.open
            orelse "meow {$_ ~~ Failure}".take and return 42;
    }() }, 'meow True'.Seq, 'sub with `with` + orelse + block interpolation';
}

When I brought up the first bug in our dev chatroom, jnthn++ pointed out that such bugs are often due to mis-scoped blocks, as p6capturelex op that's involved needs to be called in the immediate outer of the block it references.

Looking through the tickets, I also spotted skids++'s note that changing a conditional for statement_id in block migrator predicate fixed one of the tickets. This wasn't the full story of the fix, as the many still-failing tests showed, but it was a good start.

What's Your Problem?

In order to find the best solution for a bug, it's important to understand what exactly is the problem. We know mis-scoped blocks are the cause of the bug, so lets grab each of our tests, dump their QAST (--target=ast), and write out how mis-scoped the blocks are.

To make it easier to match the QAST::Blocks with the QAST::WVals referencing them, I made a modification to QAST::Node.dump to include CUID numbers and statement_id annotations in the dumps.

Going through mosts of the buggy code chunks, we have these results:

is-deeply <a b c>[$_ xx 2], <b b>.Seq, 'xx inside `with`' with 1;
# QAST for `xx` is ALONGSIDE RHS `andthen` thunk, but needs to be INSIDE

is-deeply gather {
    sub itcavuc ($c) { try {take $c} andthen 42 };
    itcavuc $_ for 2, 4, 6;
}, (2, 4, 6).Seq, 'try with block and andthen';
# QAST for try block is INSIDE RHS `andthen` thunk, but needs to be ALONGSIDE

is-deeply gather {
    sub foo ($str) { { take $str }() orelse Nil }
    foo "cc"; foo "dd";
}, <cc dd>.Seq, 'block in a sub with orelse';
# QAST for block is INSIDE RHS `andthen` thunk, but needs to be ALONGSIDE

is-deeply gather for ^7 {
    my $x = 1;
    1 andthen $x.take andthen $x = 2 andthen $x = 3 andthen $x = 4;
}, 1 xx 7, 'loop + lexical variable plus chain of andthens';
# each andthen thunk is nested inside the previous one, but all need to be
# ALONGSIDE each other

is-deeply gather for <a b c> { $^v.uc andthen $v.take orelse .say },
    <a b c>.Seq, 'loop + andthen + orelse';
# andthen's block is INSIDE orelse's but needs to be ALONGSIDE each other

is-deeply gather { (.take xx 10) given 42 }, 42 xx 10,
    'parentheses + xx + given';
# .take thunk is ALONGSIDE given's thunk, but needs to be INSIDE of it

is-deeply gather { take ("{$_}") for <aa bb> }, <aa bb>.Seq,
    'postfix for + take + block in a string';
# the $_ is ALONGSIDE `for`'s thunk, but needs to be INSIDE

is-deeply gather { take (* + $_)(32) given 10 }, 42.Seq,
    'given + whatever code closure execution';
# the WhateverCode ain't got no statement_id and is ALONGSIDE given
# block but needs to be INSIDE of it

So far, we can see a couple of patterns:

• xx and WhateverCode thunks don't get migrated, even though they should
• andthen thunks get migrated, even though they shouldn't

The first one is fairly straightforward. Looking at the QAST dump, we see xx thunk has a higher statement_id than the block it was meant to be in. This is what skids++'s hint addresses, so we'll change the statement_id conditional from == to >= to look for statement IDs higher than our current one as well, since those would be from any substatements, such as our xx inside the positional indexing operator:

($b.ann('statement_id') // -1) >= $migrate_stmt_id

The cause is very similar for the WhateverCode case, as it's missing statement_id annotation altogether, so we'll just annotate the generated QAST::Block with the statement ID. Some basic detective work gives us the location where that node is created: we search src/Perl6/Actions.nqp for word "whatever" until we spot whatever_curry method and in its guts we find the QAST::Block we want. For the statement ID, we'll grep the source for statement_id:

$ grep -FIRn 'statement_id' src/Perl6/
src/Perl6/Actions.nqp:1497:            $past.annotate('statement_id', $id);
src/Perl6/Actions.nqp:2326:                $_.annotate('statement_id', $*STATEMENT_ID);
src/Perl6/Actions.nqp:2488:                -> $b { ($b.ann('statement_id') // -1) == $stmt.ann('statement_id') });
src/Perl6/Actions.nqp:9235:                && ($b.ann('statement_id') // -1) >= $migrate_stmt_id
src/Perl6/Actions.nqp:9616:            ).annotate_self: 'statement_id', $*STATEMENT_ID;
src/Perl6/World.nqp:256:            $pad.annotate('statement_id', $*STATEMENT_ID);

From the output, we can see the ID is stored in $*STATEMENT_ID dynamic variable, so we'll use that for our annotation on the WhateverCode's QAST::Block:

my $block := QAST::Block.new(
    QAST::Stmts.new(), $past
).annotate_self: 'statement_id', $*STATEMENT_ID;

Let's compile and run our bug tests. If you're using Z-Script, you can re-compile Rakudo by running z command with no arguments:

$ z
[...]
$ ./perl6 bug-tests.t
1..1
    1..10
    ok 1 - xx inside `with`
    not ok 2 - try with block and andthen
    # Failed test 'try with block and andthen'
    # at bug-tests.t line 10
    # expected: $(2, 4, 6)
    #      got: $(2, 2, 4)
    not ok 3 - block in a sub with orelse
    # Failed test 'block in a sub with orelse'
    # at bug-tests.t line 16
    # expected: $("cc", "dd")
    #      got: $("cc", "cc")
    not ok 4 - loop + lexical variable plus chain of andthens
    # Failed test 'loop + lexical variable plus chain of andthens'
    # at bug-tests.t line 22
    # expected: $(1, 1, 1, 1, 1, 1, 1)
    #      got: $(1, 4, 3, 3, 3, 3, 3)
    not ok 5 - loop + andthen + orelse
    # Failed test 'loop + andthen + orelse'
    # at bug-tests.t line 28
    # expected: $("a", "b", "c")
    #      got: $("a", "a", "a")
    ok 6 - parentheses + xx + given
    ok 7 - postfix for + take + block in a string
    ok 8 - given + whatever code closure execution
    ok 9 - sub + given + whatevercode closure execution
    not ok 10 - sub with `with` + orelse + block interpolation
    # Failed test 'sub with `with` + orelse + block interpolation'
    # at bug-tests.t line 49
    # expected: $("meow True",)
    #      got: $("meow False",)
    # Looks like you failed 5 tests of 10
not ok 1 - thunking closure scoping
# Failed test 'thunking closure scoping'
# at bug-tests.t line 3
# Looks like you failed 1 test of 1

Looks like that fixed half of the issues already. That's pretty good!

Extra Debugging

Let's now look at the remaining failures and figure out why block migration isn't how we want it in those cases. To assists with our sleuthing efforts, let's make a couple of changes to produce more debugging info.

First, let's modify QAST::Node.dump method in NQP's repo to dump the value of in_stmt_mod annotation, by telling it to dump out the value verbatim if the key is in_stmt_mod:

if $k eq 'IN_DECL' || $k eq 'BY' || $k eq 'statement_id'
|| $k eq 'in_stmt_mod' {
    ...

Next, let's go to sub migrate_blocks in Actions.nqp and add a bunch of debug dumps inside most of the conditionals. This will let us track when a block is compared and to see whether migration occurs. As mentioned earlier, I like to key my dumps on env vars using nqp::getenvhash op, so after modifications my migrate_blocks routine looks like this; note the use of .dump method to dump QAST node guts (tip: .dump method also exists on Perl6::Grammar's match objects!):

sub migrate_blocks($from, $to, $predicate?) {
    my @decls := @($from[0]);
    my int $n := nqp::elems(@decls);
    my int $i := 0;
    while $i < $n {
        my $decl := @decls[$i];
        if nqp::istype($decl, QAST::Block) {
            nqp::atkey(nqp::getenvhash(),'ZZ') && nqp::say('ZZ1: -----------------');
            nqp::atkey(nqp::getenvhash(),'ZZ') && nqp::say('ZZ1: trying to grab ' ~ $decl.dump);
            nqp::atkey(nqp::getenvhash(),'ZZ') && nqp::say('ZZ1: to move to ' ~ $to.dump);
            if !$predicate || $predicate($decl) {
                nqp::atkey(nqp::getenvhash(),'ZZ') && nqp::say('ZZ1: grabbed');
                $to[0].push($decl);
                @decls[$i] := QAST::Op.new( :op('null') );
            }
            nqp::atkey(nqp::getenvhash(),'ZZ') && nqp::say('ZZ1: -----------------');
        }
        elsif (nqp::istype($decl, QAST::Stmt) || nqp::istype($decl, QAST::Stmts)) &&
              nqp::istype($decl[0], QAST::Block) {
            nqp::atkey(nqp::getenvhash(),'ZZ') && nqp::say('ZZ1: -----------------');
            nqp::atkey(nqp::getenvhash(),'ZZ') && nqp::say('ZZ1: trying to grab ' ~ $decl[0].dump);
            nqp::atkey(nqp::getenvhash(),'ZZ') && nqp::say('ZZ1: to move to ' ~ $to.dump);
            if !$predicate || $predicate($decl[0]) {
                nqp::atkey(nqp::getenvhash(),'ZZ') && nqp::say('ZZ1: grabbed');
                $to[0].push($decl[0]);
                $decl[0] := QAST::Op.new( :op('null') );
            }
            nqp::atkey(nqp::getenvhash(),'ZZ') && nqp::say('ZZ1: -----------------');
        }
        elsif nqp::istype($decl, QAST::Var) && $predicate && $predicate($decl) {
            $to[0].push($decl);
            @decls[$i] := QAST::Op.new( :op('null') );
        }
        $i++;
    }
}

After making the changes, we need to recompile both NQP and Rakudo. With Z-Script, we can just run z n to do that:

$ z n
[...]

Now, we'll grab the first failing code and take a look at its QAST. I'm going to use the CoreHackers::Q tool:

$ q a ./perl6 -e '
    sub itcavuc ($c) { try {say $c} andthen 42 };
    itcavuc $_ for 2, 4, 6;'
$ firefox out.html

We can see that our buggy say call lives in QAST::Block with cuid 1, which gets called from within QAST::Block with cuid 3, but is actually located within QAST::Block with cuid 2:

- QAST::Block(:cuid(3)) <wanted> :statement_id<1>
        :count<?> :signatured<?> :IN_DECL<sub>
        :in_stmt_mod<0> :code_object<?>
        :outer<?> { try {say $c} andthen 42 }
    [...]
        - QAST::Block(:cuid(2)) <wanted> :statement_id<2>
                :count<?> :in_stmt_mod<0> :code_object<?> :outer<?>
            [...]
            - QAST::Block(:cuid(1)) <wanted> :statement_id<2>
                    :IN_DECL<> :in_stmt_mod<0> :code_object<?>
                    :also_uses<?> :outer<?> {say $c}
                [...]
                - QAST::Op(call &say)  say $c
    [...]
    - QAST::Op(p6typecheckrv)
        [...]
        - QAST::WVal(Block :cuid(1))

Looks like cuid 2 block steals our cuid 1 block. Let's enable the debug env var and look at the dumps to see why exactly:

$ ZZ=1 ./perl6 -e '
    sub itcavuc ($c) { try {say $c} andthen 42 };
    itcavuc $_ for 2, 4, 6;'

ZZ1: -----------------
ZZ1: trying to grab - QAST::Block(:cuid(1)) <wanted>
    :statement_id<2> :IN_DECL<> :in_stmt_mod<0> :code_object<?>
    :also_uses<?> :outer<?> {say $c}
[...]

ZZ1: to move to - QAST::Block  :statement_id<2>
    :in_stmt_mod<0> :outer<?>

ZZ1: grabbed
ZZ1: -----------------

We can see the theft in progress. Let's take a look at our migration predicate again:

! $b.ann('in_stmt_mod')
&& ($b.ann('statement_id') // -1) >= $migrate_stmt_id

In the dump we can see in_stmt_mod is false. Were it set to a true value, the block would not be migrated—exactly what we're trying to accomplish. Let's investigate the in_stmt_mod annotation, to see when it gets set:

$ G 'in_stmt_mod' src/Perl6/Actions.nqp
2327:                $_.annotate('in_stmt_mod', $*IN_STMT_MOD);
9206:                !$b.ann('in_stmt_mod') && ($b.ann('statement_id') // -1) >= $migrate_stmt_id

$ G '$*IN_STMT_MOD' src/Perl6/Grammar.nqp
1200:        :my $*IN_STMT_MOD := 0;                    # are we inside a statement modifier?
1328:        :my $*IN_STMT_MOD := 0;
1338:        | <EXPR> :dba('statement end') { $*IN_STMT_MOD := 1 }

Looks like it's a marker for statement modifier conditions. Statement modifiers have a lot of relevance to our andthen thunks, because $foo with $bar gets turned into $bar andthen $foo during parsing. Since, as we can see in src/Perl6/Grammar.nqp, in_stmt_mod annotation gets set for with statement modifiers, we can hypothesize that if we turn our buggy andthen into a with, the bug will disappear:

$ ./perl6 -e 'sub itcavuc ($c) { 42 with try {say $c} };
    itcavuc $_ for 2, 4, 6;'
2
4
6

And indeed it does! Then, we have a way forward: we need to set in_stmt_mod annotation to a truthy value for just the first argument of andthen (and its relatives notandthen and orelse).

Glancing at the Grammar it doesn't look like it immediatelly offers a similar opportunity for how in_stmt_mod is set for the with statement modifier. Let's approach it differently. Since we care about this when thunks are created, let's watch for andthen QAST inside sub thunkity_thunk in Actions, then descend into its first kid and add the in_stmt_mod annotation by cheating and using the past_block annotation on QAST::WVal with the thunk that contains the reference to QAST::Block we wish to annotate. The code will look something like this:

sub mark_blocks_as_andnotelse_first_arg($ast) {
    if $ast && nqp::can($ast, 'ann') && $ast.ann('past_block') {
        $ast.ann('past_block').annotate: 'in_stmt_mod', 1;
    }
    elsif nqp::istype($ast, QAST::Op)
    || nqp::istype($ast, QAST::Stmt)
    || nqp::istype($ast, QAST::Stmts) {
        mark_blocks_as_andnotelse_first_arg($_) for @($ast)
    }
}

sub thunkity_thunk($/,$thunky,$past,@clause) {
    [...]

    my $andnotelse_thunk := nqp::istype($past, QAST::Op)
      && $past.op eq 'call'
      && ( $past.name eq '&infix:<andthen>'
        || $past.name eq '&infix:<notandthen>'
        || $past.name eq '&infix:<orelse>');

    while $i < $e {
        my $ast := @clause[$i];
        $ast := $ast.ast if nqp::can($ast,'ast');
        mark_blocks_as_andnotelse_first_arg($ast)
            if $andnotelse_thunk && $i == 0;
        [...]

First, we rake $past argument given to thunkity_thunk for a QAST::Op for nqp::call that calls one of our ops—when we found one, we set a variable to a truthy value. Then, in the loop, when we're iterating over the first child node ($i == 0) of these ops, we'll pass its QAST to our newly minted mark_blocks_as_andnotelse_first_arg routine, inside of which we recurse over any ops that can have kids and mark anything that has past_block annotation with truthy in_stmt_mod annotation.

Let's compile our concoction and give the tests another run. Once again, I'm using Z-Script to recompile Rakudo:

$ z
[...]
$ ./perl6 bug-tests.t
1..1
    1..10
    ok 1 - xx inside `with`
    ok 2 - try with block and andthen
    ok 3 - block in a sub with orelse
    not ok 4 - loop + lexical variable plus chain of andthens
    # Failed test 'loop + lexical variable plus chain of andthens'
    # at bug-tests.t line 23
    # expected: $(1, 1, 1, 1, 1, 1, 1)
    #      got: $(1, 4, 3, 3, 3, 3, 3)
    ok 5 - loop + andthen + orelse
    ok 6 - parentheses + xx + given
    ok 7 - postfix for + take + block in a string
    ok 8 - given + whatever code closure execution
    ok 9 - sub + given + whatevercode closure execution
    not ok 10 - sub with `with` + orelse + block interpolation
    # Failed test 'sub with `with` + orelse + block interpolation'
    # at bug-tests.t line 50
    # expected: $("meow True",)
    #      got: $("meow False",)
    # Looks like you failed 2 tests of 10
not ok 1 - thunking closure scoping
# Failed test 'thunking closure scoping'
# at bug-tests.t line 4
# Looks like you failed 1 test of 1

We got closer to the goal, with 80% of the tests now passing! In the first remaining failure, we already know from our original examination that chained andthen thunks get nested when they should not—we haven't done anything to fix that yet. Let's take care of that first.

Playing Chinese Food Mind Games

Looking back out at the fixes we applied already, we have a marker for when we're working with andthen or its sister ops: the $andnotelse_thunk variable. It seems fairly straight-forward that if we don't want the thunks of these ops to migrate, we just need to annotate them appropriately and stick the check for that annotation into the migration predicate.

In Grammar.nqp, we can see our ops are configured with the .b thunky, so we'll locate that branch in sub thunkity_thunk and pass $andnotelse_thunk variable as a new named param to the make_topic_block_ref block maker:

...
elsif $type eq 'b' {  # thunk and topicalize to a block
    unless $ast.ann('bare_block') || $ast.ann('past_block') {
        $ast := block_closure(make_topic_block_ref(@clause[$i],
          $ast, :$andnotelse_thunk,
          migrate_stmt_id => $*STATEMENT_ID));
    }
    $past.push($ast);
}
...

The block maker) will shove it into the migration predicate, so our block maker code becomes this:

 sub make_topic_block_ref(
    $/, $past, :$copy, :$andnotelse_thunk, :$migrate_stmt_id,
 ) {
    my $block := $*W.push_lexpad($/);

    # Add annotation to thunks of our ops:
    $block.annotate: 'andnotelse_thunk', 1 if $andnotelse_thunk;

    $block[0].push
        QAST::Var.new( :name('$_'), :scope('lexical'), :decl('var') );
    $block.push($past);
    $*W.pop_lexpad();
    if nqp::defined($migrate_stmt_id) {
        migrate_blocks($*W.cur_lexpad(), $block, -> $b {
               ! $b.ann('in_stmt_mod')

            # Don't migrate thunks of our ops:
            && ! $b.ann('andnotelse_thunk')

            && ($b.ann('statement_id') // -1) >= $migrate_stmt_id
        });
    }
    ...

One more compilation cycle and test run:

$ z
[...]
$ ./perl6 bug-tests.t
1..1
    1..10
    ok 1 - xx inside `with`
    ok 2 - try with block and andthen
    ok 3 - block in a sub with orelse
    ok 4 - loop + lexical variable plus chain of andthens
    ok 5 - loop + andthen + orelse
    ok 6 - parentheses + xx + given
    ok 7 - postfix for + take + block in a string
    ok 8 - given + whatever code closure execution
    ok 9 - sub + given + whatevercode closure execution
    not ok 10 - sub with `with` + orelse + block interpolation
    # Failed test 'sub with `with` + orelse + block interpolation'
    # at bug-tests.t line 50
    # expected: $("meow True",)
    #      got: $("meow False",)
    # Looks like you failed 1 test of 10
not ok 1 - thunking closure scoping
# Failed test 'thunking closure scoping'
# at bug-tests.t line 4
# Looks like you failed 1 test of 1

So close! Just a single test failure remains. Let's give it a close look.

Within and Without

Let's repeat our procedure of dumping QASTs as well as enabing the ZZ env var and looking at what's causing the thunk mis-migration. I'm going to run a slightly simplified version of the failing test, to keep the cruft out of QAST dumps. If you're following along, when looking at full QAST dump keep in mind what I mentioned earlier: with gets rewritten into andthen op call during parsing.

$ q a ./perl6 -e '.uc with +"a" orelse "meow {$_ ~~ Failure}".say and 42'
$ firefox out.html

- QAST::Block(:cuid(4)) :in_stmt_mod<0>
    [...]
    - QAST::Block(:cuid(1))  :statement_id<1> :in_stmt_mod<1>
      [...]
      - QAST::Op(chain &infix:<~~>) <wanted> :statement_id<2> ~~
        - QAST::Var(lexical $_) <wanted> $_
        - QAST::WVal(Failure) <wanted> Failure
    - QAST::Block(:cuid(2)) :statement_id<1>
        :in_stmt_mod<1> :andnotelse_thunk<1>
      [...]
      - QAST::Op(callmethod Stringy) <wanted>
        - QAST::Op(call) <wanted> {$_ ~~ Failure}
          - QAST::Op(p6capturelex) <wanted> :code_object<?>
            - QAST::Op(callmethod clone)
              - QAST::WVal(Block)

$ ZZ=1 ./perl6 -e '.uc with +"a" orelse "meow {$_ ~~ Failure}".say and 42'
[...]
ZZ1: -----------------
ZZ1: trying to grab - QAST::Block(:cuid(1))
  :statement_id<1> :in_stmt_mod<1>
  [...]
ZZ1: to move to - QAST::Block
  :statement_id<1> :andnotelse_thunk<1> :in_stmt_mod<1>
  [...]
ZZ1: -----------------

Although QAST::WVal lacks .past_block annotation and so doesn't show the block's CUID in the dump, just by reading the code dumped around that QAST, we can see that the CUID-less block is our QAST::Block :cuid(1), whose immediate outer is QAST::Block :cuid(4), yet it's called from within QAST::Block :cuid(2). It's supposed to get migrated, but that migration never happens, as we can see when we use the ZZ env var to enable our debug dumps in the sub migrate_blocks.

We can see why. Here's our current migration predicate (where $b is the examined block, which in our case is QAST::Block :cuid(1)):

   ! $b.ann('in_stmt_mod')
&& ! $b.ann('andnotelse_thunk')
&& ($b.ann('statement_id') // -1) >= $migrate_stmt_id

The very first condition prevents our migration, as our block has truthy in_stmt_mod annotation, because it's part of the with's condition. At the same time, it does need to be migrated because it's part of the andthen thunk that's inside the statement modifier!

Since we already have $andnotelse_thunk variable in the vicinity of the migration predicate we can use it to tell us whether we're migrating for the benefit of our andthen thunk and not the statement modifier. However, recall that we've used the very same in_stmt_mod annotation to mark the first argument of andthen and its brother ops. We need to alter that first.

And so, the sub mark_blocks_as_andnotelse_first_arg we added earlier becomes:

sub mark_blocks_as_andnotelse_first_arg($ast) {
    if $ast && nqp::can($ast, 'ann') && $ast.ann('past_block') {
        $ast.ann('past_block').annotate: 'in_stmt_mod_andnotelse', 1;
    }
    ...

And then we tweak the migration predicate to watch for this altered annotation and to consider the value of $andnotelse_thunk variable:

migrate_blocks($*W.cur_lexpad(), $block, -> $b {
    (    (! $b.ann('in_stmt_mod_andnotelse') &&   $andnotelse_thunk)
      || (! $b.ann('in_stmt_mod')            && ! $andnotelse_thunk)
    )
    && ($b.ann('statement_id') // -1) >= $migrate_stmt_id
    && ! $b.has_ann('andnotelse_thunk')
});

Thus, we migrate all the blocks with statement_id equal to or higher than ours and are all of the following:

• Not thunks of actual andthen, notandthen, or orelse
• Not thunks inside a statement modifier, unless they're inside thunks of andthen or related ops
• If we're considering migrating them inside one of the andthen's thunks, then also not part of the first argument to andthen (or related ops), .

That's a fancy-pants predicate. Let's compile and see if it gets the job done:

$ z
[...]
$ ./perl6 bug-tests.t
  1..1
    1..10
    ok 1 - xx inside `with`
    ok 2 - try with block and andthen
    ok 3 - block in a sub with orelse
    ok 4 - loop + lexical variable plus chain of andthens
    ok 5 - loop + andthen + orelse
    ok 6 - parentheses + xx + given
    ok 7 - postfix for + take + block in a string
    ok 8 - given + whatever code closure execution
    ok 9 - sub + given + whatevercode closure execution
    ok 10 - sub with `with` + orelse + block interpolation
ok 1 - thunking closure scoping

Success! Now, let's remove all of the debug statements we added. Then, recompile and run make stresstest, to ensure we did not break anything else. With Z-Script, we can do all that by just running z ss:

$ z ss
[...]
All tests successful.
Files=1287, Tests=153127, 159 wallclock secs (21.40 usr  3.27 sys + 3418.56 cusr 179.32 csys = 3622.55 CPU)
Result: PASS

All is green. We can now commit our fix to Rakudo's repo, then commit our tests to the roast repo, and all that remains is closing those 10 tickets we fixed!

Job well done.

Conclusion

Today, we learned quite a bit about QAST: the Abstract Syntax Trees Perl 6 code compiles to in the Rakudo compiler. We examined the common types of QAST and how to create, annotate, mutate, execute, and dump them for examination.

In the second part of the article, we applied our new knowledge to fix a hive of mis-scoped thunking bugs that plagued various Perl 6 constructs. We introspected the generated QAST nodes to specially annotate them, and then used those annotations to reconfigure migration predicate, so that it migrates the blocks correctly.

Hopefully, this knowledge inspires you to fix the many other bugs we have on the RT tracker as well as our GitHub Issue tracker

-Ofun

You probably have read a recent Open Letter to the Perl Community. The letter has generated a lot of response (173 reddit comments as I write this). Unfortunately, a lot of the responses are quite negative and do not match my understanding of the letter.

I figured I share my interpretation of the words and if that interpretation does not match the author's intent, I hope my interpretation captures the mood of the community on the matter.

A Radical Idea

The first thing the letter talks about is what it calls a "radical idea". The suggestion is for Perl 5 compiler to upgrade its backend. Instead of the current Perl VM the perl compiler uses, it would be ported to NQP compiler toolkit and will support all of the backends NQP supports, which currently is MoarVM, as the leading option, with JVM and JavaScript backends at various stages of completion as alternatives.

Perl 5 the language would remain as is, but the new backend would allow the compiler to make use of modern features such as JIT compilation and good threading support. So to clarify, this wasn't a suggestion to port Perl 5 to Perl 6, but merely to swap the backend Perl 5 compiler uses.

Perl 5 Porters view this radical idea as a non-starter at the moment, so I think we can safely bury it and move on to the second idea the letter talks about

Porting Modules

As I write this, the Perl 6 ecosystem is 8 modules short of 1000 modules total. A lot of the needed features are currently missing native Perl 6 implementation and many users choose to use Perl 5's CPAN ecosystem via Inline::Perl5 module.

While Inline::Perl5 works rather well, it's not an ideal option for many developers. Thus, the Letter's author proposed an incentive for people to port much-needed Perl 5 modules to Perl 6 in the form of a leaderboard and potential support from sponsors who would donate to Perl 6 Core development fund in exchange for ported modules.

So to clarify, the goal behind the porting of modules is merely to beef up Perl 6's ecosystem and is not an implication that Perl 5 users have to or even might want to move to any other language.

The Elephant in The Room

While not mentioned in the original Letter, a frequent theme in the comments was that Perl 6 should be renamed, as the name is inaccurate or is damaging.

This is the topic on which I wrote more than once and those who have been following closely know that, yes, many (but by no means all) in the Perl 6 community acknowledge the name is detrimental to both Perl 6 and Perl 5 projects.

This is why with a nod of approval from Larry we're moving to create an alias to Perl 6 name during 6.d language release, to be available for marketing in areas where "Perl 6" is not a desirable name.

While some may feel an alias—as opposed to a full rename—is not quite what they desired, I believe if the alternate name is as beneficial as the proponents of the rename believe it to be, the alias will naturally be able to overtake "Perl 6" name in usage and become the primary name through sheer amount of use. Thus, the alias can be seen as the chance for the rename crowd to prove their claims.

Long Live Perl 5!

Perl 6 isn't a desirable language for many Perl 5 users, for various reasons. That's perfectly valid and is understandable. Perl 6 is quite different from its sister language and the choice to switch to it isn't much different than choosing whether to switch to Ruby, Go, Rust, or any other of the multitude of languages available.

Since Perl 5 is actively maintained, its users should not feel compelled to migrate to anything else, and should continue using what works for them, regardless of what opinions members of other programming communities might hold.

Conclusion

The Letter suggested a radical idea that the perl compiler's backend be ported from Perl VM to NQP that supports multiple modern VMs. Based on the response to the proposal, it does not appear to be viable at the current time. The more prominent suggestion in the Letter is the call to port Perl 5 modules to Perl 6, to help Perl 6 users get native-Perl 6 access to valuable Perl 5 modules.

While there are Perl 6 users who believe Perl 6 is superiour to Perl 5, it does not mean Perl 5 users should feel compelled to upgrade to anything but the latest stable version of Perl 5.

I hope that regardless of which flavour of Perl we choose to use and call our favourite, we can unite and exist together as The Perl Community.

Read this article on Rakudo.Party

Development of Rakudo Perl 6 is quite fast-paced, with hundreds of commits made each month to its five core repositories. Users undoubtedly feel some impact from those commits: bug fixes may break code that relied on them, backend changes may have unforeseen impact on the user code, new useful features may be implemented that users would want to know about.

In the past, for things with very large impact, we made blog posts, but there are lots of small things that fly under the radar, unless you actively pay a lot of attention to Rakudo Perl 6's core development.

To help all of our users to be aware of important issues, we're announcing introduction of P6lert service: tweet-sized alerts from Perl 6 Core Developers.

The Goods

The P6lert service primarily consists of alerts.perl6.org website, but with it come a variety of ways to receive alerts posted on it:

• All alerts are automatically tweeted by unmonitored @p6lert Twitter account; all tweets include #p6lert hashtag
• There is an RSS feed you can subscribe to
• The site offers an API, which you can use in Perl 6 via WWW::P6lert module.
• Command line p6lert script gets you access to alerts from command line and can even be integrated into compiler upgrade scripts.
• An embeddable HTML Widget can show alerts on your blog or any other useful place.

The Content

While we'll make adjustments as we move forward, I foresee most of the non-critical alerts will largely include things that are: (a) more important than simply hoping users-who-care will read about it in the ChangeLog; (b) not as important to warrant a notification blog post.

As a rule-of-thumb, if you picture a user who added p6lert script to their compiler upgrade procedure, the alerts the script will show will inform that user on everything they need to know to perform that upgrade safely.

The alerts are also deliberately length-limited to be easy to process and fast to digest. As they're posted via an IRC bot, the poster has at most about 400 characters to work with.

Each alert has an affects field for it, giving additional info what the alert applies to. I think it'll often be empty, as alerts affecting latest compiler versions imply they affect whatever latest release is at the time alert was posted.

The alerts have a severity rating: low, normal, and high indicating how important an alert is. Along with those, come two out-of-band ratings: info and critical. Info alerts will usually be something the users don't need to act upon, while critical alerts will often simply contain a link to a blog post that details a critical issue.

Of the top of my head, here are some real-life examples from the past and how I'd rate their severity on the P6lert service:

• info - Telemetry implementation. This was a fairly large implementation of a feature in the core and some users may be interested in it. At the same time, they don't have to act on this alert. Rakudo and Rakudo Star release may also be info alert material.
• low - implementation of output buffering on IO handles. Once that was implemented, we noticed some minor fallout in code that assumed lack of buffering. A low-severity alert could notify users about this.
• normal - A real-life normal-severity alert is already posted on the site. During 6.d spec pre-release review, the Str.parse-names method was placed under deprecation and its functionality was moved to live under Str.uniparse. While this method always existed as a 6.d-proposal, it's known to us that some users already use it and an alert will help them ensure their code keeps working past 6.e language release.
• high - a while ago, .subst was briefly made to die if it couldn't write to $/; same as some methods already do. Upon examination of ecosystem fallout, this change was reverted, pending further review. However, were it to stay, a high-severity alert would be in order.
• critical - when we finished work on lexical require, conditional loading of modules many users used was silently failing and the users needed to change their code to correct reliance on the old, buggy behavour. We issued a blog post with upgrade instructions. A critical alert would be a link to this blog post.

That's my vision for host the system will be used, but it'll evolve to suit our needs as more core devs and more of our users start using it. The core dev docs for the system along with code for all the pieces is available in perl6/alerts repo.

Conclusion

As part of improving using user experience, Perl 6 core devs now offer alerts.perl6.org service that will list important information about latest developments in the land of Perl 6. There are numerous ways to consume those alerts, such as an RSS or Tweeter feeds, a command line utility, and an easy-to-use API.

The alerts will come in 5 different severity ratings, indicating their importance. We'll continue to improve the system to best suit our users' needs.

If you have any questions or feedback, you can always talk to the core devs on #perl6-dev IRC chat.

-Ofun

Every year since 2009, the Rakudo Perl 6 community publishes a Rakudo Perl 6 advent calendar, in the form of blog posts on perl6advent.wordpress.com.

To keep up this great tradition, we need 24 blog posts, and volunteers who write them. If you want to contribute a blog post about anything related to Rakudo Perl 6, please add your name (and potentially also a topic already) to the schedule, and if you don't yet have a login on the advent blog, please tell Zoffix or someone on #perl6 IRC chat your email address so that they can send you an invitation.

Rakudo Perl 6 advent blog posts should be finished the day before they are due, and published with midnight (UTC) of the due date as publishing date.

If you have any questions, or want to discuss blog post ideas, please join on the #perl6 IRC channel on irc.freenode.org.

Read this article on Rakudo.Party

If you've been following Rakudo's development since first language release on Christmas, 2015, you might've heard of numerous people working to bring CPAN support to Rakudo Perl 6.

Good news! It's finally here in usable form and you should start using it!

Let's talk about all the moving parts and how to upload your dists to CPAN.

The Moving Parts and Status Report

All of the heavy lifting has been done awhile back, during Perl Toolchain Summit and other times. I wasn't present for it to know the details, but to catch up you could join #perl6-toolchain chat and talk to humans or read the channel log. PAUSE/CPAN support for Perl 6 dists was implemented and zef module installer was trained to check for CPAN dists as well as our GitHub/GitLab-based ecosystem (called "p6c").

The only bit that was left missing is a front-end to browse available CPAN dists. There is a team who wished to take metacpan.org's codebase and modify it for Rakudo dists. I'm told that project is currently "stalled but not dead".

That's unfortunate, however, earlier this week, modules.perl6.org was taught to handle CPAN dists, so—hooray!—we finally have some sort of a front-end for CPAN dists. If you only want to see CPAN dists in search results, you can use from:cpan search qualifier (just like you can use from:github and from:gitlab ones).

GitHub/GitLab dists URLs still direct to repos, but CPAN dists have a file browser that lets you see what files are up in the dist. The file browser also renders README.md/README.markdown markdown readme files.

The viewer doesn't have all the bells and whistles of metacpan.org and doesn't (yet) render POD6, but it's certainly useable. The person who implemented this viewer will be busy preparing 6.d language release in the near future and won't have the time to make additional improvements to the CPAN dist viewer. So… you're invited to contribute and make it better!

Why Upload to CPAN

CPAN has many mirrors ensuring module installation is not affected whenever GitHub (a single website) has issues. The uploaded dists are also immutable and stay there forever (barring special deletion requests, even deleted dists remain available on BackPAN). This means people are more likely to trust these dists for use in their larger projects that need dependable dependencies. Lastly… it's what the cool kids use!

How to Upload to CPAN

Here's the process for how you can get your dists to CPAN. If these dists are currently listed in our p6c ecosystem, both p6c and CPAN versions will appear on modules.perl6.org, and you're encouraged to remove the p6c version. Some of the described tools are brand-new and others are brand-old, created before Rakudo existed, so treat this guide as part information and part invitation to improve the tools.

Step 1: Get a PAUSE account

PAUSE stands for "The [Perl programming] Authors Upload Server", it's located at pause.perl.org, and it's a site you use to upload dists to CPAN.

Go to request PAUSE account page and subscribe for an account. The "desired ID" field is for your PAUSE ID, and it's currently used as "author" field on modules.perl6.org. For example, mine is ZOFFIX.

I had my account for over a decade, so my memory is a bit fuzzy, but I think you'll need to wait for a human to approve and create your account—it's not instantaneous.

Step 2: Make a Dist Archive

You can manually create a tarball or a zip archive. I don't have all the details on which files you're supposed to have in them; you can take a look at other CPAN dists to see what they're doing or…

Use App::Mi6 module! It's possible you were already using it to create dists, in which case you're in luck, as you can just run mi6 dist to make a dist archive.

I rolled my dists by hand and wrote all the docs in README.md, so when I gave mi6 dist a whirl, it replaced my README.md with emptiness because I wasn't using any POD6—something (currently) to watch out for.

Step 3: Upload Your Dist

The first option is to upload manually: log into pause.perl.org, then go to Upload a file to CPAN, be sure to select Perl6 in the select input and then upload either via an uploaded file or a URL.

The second option is to use App::Mi6's mi6 upload command.

Shortly after the upload, you'll get an email about whether your upload succeeded (you can also see emails on nntp.perl.org). Make sure you have a META6.json file in your dist and that the dist version you're uploading is higher than the currently uploaded version. Those are the most common upload errors.

Step 4: Relax and Wait

If you're on IRC, in about 10 minutes after your upload, our buggable robot will announce it:

<buggable> New CPAN upload: Number-Denominate-1.001001.tar.gz by ZOFFIX
    https://www.cpan.org/authors/id/Z/ZO/ZOFFIX/Perl6/Number-Denominate-1.001001.tar.gz

In about 2 hours, the dist will also appear on modules.perl6.org. Its updater is started in a cron job on 20th and 40th minute of the hour (unless a job is already running) and it takes about 2 hours to finish each run.

Step 5: Celebrate with the Appropriate Amount of Fun

That's about it to the process. I foresee more tools will be created in the future to make the process even easier than it is today. If you have any questions or issues, just talk to a human or a robot on our #perl6 IRC channel.

Conclusion

CPAN support for Rakudo Perl 6 dists is now usably here. You're encouraged to upload your dists to CPAN, to grow a more dependable ecosystem. You're also invited to improve and create tooling that manages and displays CPAN uploads.

-Ofun

Read this article on 6lang.Party

When a couple months ago I rekindled the naming debate—the discussion on whether "Perl 6" should be renamed—I didn't expect anything more than a collective groan. That wasn't the case and today, I figured, I'd post a progress report and list the salient happenings, all the way to my currently being the proud owner of 6lang.party domain name.

The "Rakudo" Language

The "new" name I mentioned in my original post was Rakudo. As many quickly pointed out, it wasn't the greatest of names because it was the name of an implementation. Yes, I agree, but originally I thought few, if any, would be on board with a new name, or extended name, and Rakudo was basically the only name people already were using, so it stood out as something that could be "hijacked."

The Blog Post Fallout

There was quite a bit of discussion on r/perl, r/perl6, and blogs.perl.org. The general mood among the Perl community members who aren't avid 6lang users was that the entirely new name was a good idea. However, the 6lang users, and especially core devs, overall, argued "Perl 6" still had some recognition benefits and should not be removed entirely.

The middle ground was aimed at then: extend the language name. The "official" name would be among the lines of "Blah Perl 6" and users opposed to the 4-letter swear word would just use the name extension on its own, while those who feel the original name has benefits can still reap them.

The decision on the naming extension was placed on the 6.d language release agenda, with the final call on whether and with what the name should to be extended to be done by Larry, when we cut the 6.d language release.

The 6lang

Fast-forward two months. A kind soul (thank you, by the way!) asked Larry what he thought about the naming debate during the last Perl Conference:

Larry opined that we could have other terms by which Perl versions or Perl distributions are marketed as. So that gives us an option to pick an alternative name to be the second name with any "official" standing. Personally, I really like this idea; even more than name extension, because should there indeed be more benefit to the name without "Perl" in it, the alternative name will naturally become the most-used one.

Another core dev, AlexDaniel++, coined an alternative name: spelt 6lang; can be pronounced as slang, if you want to be fancy. I really liked the name, so I jumped in and registered 6lang.party

<AlexDaniel> Zoffix++ for making me recognize the need for
     alternative name. For a long time I was against
<AlexDaniel> and honestly, I can start using something like 6lang
     right away. “Rakudo Perl 6” is infringing on
     language/compiler distinction so I'm feeling reluctant
<Zoffix> OK, I'll too start using 6lang
* Zoffix is now a proud owner of 6lang.party :D
<timotimo> wow
<AlexDaniel> that was quick

And a couple of hours later, our Marketing Department churned out a new poster:

The drawback is that the name can't be used as an identifier… and Larry doesn't think it's a terribly sexy name.

* TimToady notes that 6lang isn't gonna work anywhere an identifier
     needs a leading alpha
<TimToady> it's also not a terribly sexy name
<TimToady> I could go for something more like psix, "where the p is silent
     if you want it to be" :)

Although, on the plus side, the name has the benefit that alphabetically it sorts earlier than pretty much any other language.

<AlexDaniel> If we see “6lang” as a more marketable alternative, then
     the fact that some things may not parse it as an identifier
     practically does not matter. However, this little bit is quite useful:
<AlexDaniel> m: <perl5 golang c# 6lang ruby>.sort.say
<camelia> rakudo-moar 39a4b7: OUTPUT: «(6lang c# golang perl5 ruby)␤»
<AlexDaniel> :)
<AlexDaniel> .oO( AAAlang – batteries included )

To 6.d Release And Beyond

So that's where things progressed to so far. No official decisions have been made yet, but we're thinking about it and playing with the idea. The decision on the naming debate is to be made during 6.d release.

Having learned a painful lesson from The Christmas release, we're reluctant to put down any dates for 6.d release, but I suspect it'll be somewhere between the upcoming New Year's and It's-Ready-When-It's-Ready.

See you then \o

Read this article on Rakudo.Party

When I first joined the Rakudo project, we used to say "there are none right now; check back in a year" whenever someone asked for a book about the language. Today, there's a whole website for picking out a book, and the number of available books seems to multiply every time I look at it.

Still, I feel something is amiss, when I talk to folks on our support chat, when I read blog posts about the language, or when I look at our official language documentation. And it's due to that feeling that I wish to join the Rakudo book-writing club and write a few of my own. I dub it: The Rakudo Book Project.

The Books

The Rakudo Book Project involves 3 main books—The White Book, The Gray Book, and The Black Book—as well as 2 half-books—The Green Book and The Cracked Book.

The White Book will aim to provide introductory material to the Rakudo language. The target audience will benefit from prior programming experience, but it won't be strictly necessary for computer-savy people. The target audience is "adept beginners", as some might call it.

The book will cover most of Rakudo's features a typical Rakudo programmer might use in their projects, but it won't cover every little thing about each of them. By the end of the book, the readers will have written several programming projects and will be comfortable making useful, real-world Rakudo programs. More in-depth coverage of the language will be provided by The Gray Book, which is what The White Book's readers would read next. The Black Book will reach even deeper, exploring all of the arcane constructs. The progression through the books can be thought of as a plant growing in a flower pot. Initially, the roots extend through a large area of the pot, but they don't go all the way to all the walls and are rather sparse. As the plant grows, more and more roots shoot out, covering more and more volume of the pot. Same is with the books; while reading The White Book alone will let the plant survive, the root coverage will be sparse. However, by the end of The Black Book, the reader will be an expert Rakudo programmer.

Those three books are the core of my planned project. They're supplemented by two half-books on each end of the knowledge spectrum. The Green Book will target absolute programming beginners and get them up to speed just enough so they would be able to comfortably continue their learning using The White Book. On the other end of the spectrum is The Cracked Book. It's a half-book that follows The Black Book and won't provide more advanced techniques per say, but rather arcane "hacks" or even "bad ideas" that one might not wish to use in real-life code but which nevertheless provide some insight into the language.

The Cracked Book is yet a faint glimmer of an idea. Whether it will actually be made will depend on how much more I will want to say after The Black Book is complete. The Green Book is currently a bit amorphous as well. I have a 12-year old sibling interested in computers, so The Green Book might end up being a Rakudo For Kids.

The likely order in which the books will be produced is White, Gray, Green, Black, and Cracked. It's an ambitious plan, and so I won't be making any promises for producing more than one book at a time. Thus, the current aim is to produce just The White Book.

The Price

The digital versions of the books will be available for free.

Since Rakudo development can always use more funding, I plan to run crowd-funding campaigns during each of the book's development. 100% of all the collected funds will be used to sponsor Rakudo work (sponsoring someone other than me, of course). The campaigns will start once half of the target book has been created and the backers will get early preview digital copies as the book is developed further, as well as honourable mentions as Rakudo sponsors in the book itself.

Thus, the first Rakudo Core Fundraiser will launch once I have the first half of The White Book finished. I'm hoping that will happen soon.

The Why

Other than the obvious reason why people write the books—giving an alternate take on the material—I'd like to do this to cross off an item off my bucket list. Having written a terrible non-fiction book, lackluster fiction book, and a decent illustrated children's book, I hope to add a great technical book to the list, to complete it. I figure, with 5 books to attempt it, I'll be successful.

As for my alternate take, I hope to squash the myth that Rakudo is too big to learn as well as carve out a well-defined path for learners to follow. Just as I could make a living 10 years ago, when I barely spoke English, so a beginner Rakudo programmer can make useful programs with rudimentary knowledge of the language. The key is to not try to learn everything at once as well as have a definite path to walk through. Hence the 5 separate books.

I'm hoping at the end of this journey I will have accomplished all of these goals.

See you at the first Rakudo Core Fundraiser.

Read this article on Perl6.Party

This is the second part in the series! Be sure you read Part I first where we discuss what Seqs are and how to .cache them.

Today, we'll take the Seq apart and see what's up in it; what drives it; and how to make it do exactly what we want.

PART II: That Iterated Quickly

The main piece that makes a Seq do its thing is an object that does the Iterator role. It's this object that knows how to generate the next value, whenever we try to pull a value from a Seq, or push all of its values somewhere, or simply discard all of the remaining values.

Keep in mind that you never need to use Iterator's methods directly, when making use of a Seq as a source of values. They are called indirectly under the hood in various Perl 6 constructs. The use case for calling those methods yourself is often the time when we're making an Iterator that's fed by another Iterator, as we'll see.

Pull my finger...

In its most basic form, an Iterator object needs to provide only one method: .pull-one

my $seq := Seq.new: class :: does Iterator {
    method pull-one {
        return $++ if $++ < 4;
        IterationEnd
    }
}.new;

.say for $seq;

# OUTPUT:
# 0
# 1
# 2
# 3

Above, we create a Seq using its .new method that expects an instantiated Iterator, for which we use an anonymous class that does the Iterator role and provides a single .pull-one method that uses a pair of anonymous state variables to generate 4 numbers, one per call, and then returns IterationEnd constant to signal the Iterator does not have any more values to produce.

The Iterator protocol forbids attempting to fetch more values from an Iterator once it generated the IterationEnd value, so your Iterator's methods may assume they'll never get called again past that point.

Meet the rest of the gang

The Iterator role defines several more methods, all of which are optional to implement, and most of which have some sort of default implementation. The extra methods are there for optimization purposes that let you take shortcuts depending on how the sequence is iterated over.

Let's build a Seq that hashes a bunch of data using Crypt::Bcryptmodule (run zef install Crypt::Bcrypt to install it). We'll start with the most basic Iterator that provides .pull-one method and then we'll optimize it to perform better in different circumstances.

use Crypt::Bcrypt;

sub hash-it (*@stuff) {
    Seq.new: class :: does Iterator {
        has @.stuff;
        method pull-one {
            @!stuff ?? bcrypt-hash @!stuff.shift, :15rounds
                    !! IterationEnd
        }
    }.new: :@stuff
}

my $hashes := hash-it <foo bar ber>;
for $hashes {
    say "Fetched value #{++$} {now - INIT now}";
    say "\t$_";
}

# OUTPUT:
# Fetched value #1 2.26035863
#     $2b$15$ZspycxXAHoiDpK99YuMWqeXUJX4XZ3cNNzTMwhfF8kEudqli.lSIa
# Fetched value #2 4.49311657
#     $2b$15$GiqWNgaaVbHABT6yBh7aAec0r5Vwl4AUPYmDqPlac.pK4RPOUNv1K
# Fetched value #3 6.71103435
#     $2b$15$zq0mf6Qv3Xv8oIDp686eYeTixCw1aF9/EqpV/bH2SohbbImXRSati

In the above program, we wrapped all the Seq making stuff inside a sub called hash-it. We slurp all the positional arguments given to that sub and instantiate a new Seq with an anonymous class as the Iterator. We use attribute @!stuff to store the stuff we need to hash. In the .pull-one method we check if we still have @!stuff to hash; if we do, we shift a value off @!stuff and hash it, using 15 rounds to make the hashing algo take some time. Lastly, we added a say statement to measure how long the program has been running for each iteration, using two now calls, one of which is run with the INIT phaser. From the output, we see it takes about 2.2 seconds to hash a single string.

Skipping breakfast

Using a for loop, is not the only way to use the Seq returned by our hashing routine. What if some user doesn't care about the first few hashes? For example, they could write a piece of code like this:

my $hash = hash-it(<foo bar ber>).skip(2).head;
say "Made hash {now - INIT now}";
say bcrypt-match 'ber', $hash;

# OUTPUT:
# Made hash 6.6813790
# True

We've used Crypt::Bcryptmodule's bcrypt-match routine to ensure the hash we got matches our third input string and it does, but look at the timing in the output. It took 6.7s to produce that single hash!

In fact, things will look the worse the more items the user tries to skip. If the user calls our hash-it with a ton of items and then tries to .skip the first 1,000,000 elements to get at the 1,000,001^st hash, they'll be waiting for about 25 days for that single hash to be produced!!

The reason is our basic Iterator only knows how to .pull-one, so the skip operation still generates the hashes, just to discard them. Since the values our Iterator generates do not depend on previous values, we can implement one of the optimizing methods to skip iterations cheaply:

use Crypt::Bcrypt;

sub hash-it (*@stuff) {
    Seq.new: class :: does Iterator {
        has @.stuff;
        method pull-one {
            @!stuff ?? bcrypt-hash @!stuff.shift, :15rounds
                    !! IterationEnd
        }
        method skip-one {
            return False unless @!stuff;
            @!stuff.shift;
            True
        }
    }.new: :@stuff
}

my $hash = hash-it(<foo bar ber>).skip(2).head;
say "Made hash {now - INIT now}";
say bcrypt-match 'ber', $hash;

# OUTPUT:
# Made hash 2.2548012
# True

We added a .skip-one method to our Iterator that instead of hashing a value, simply discards it. It needs to return a truthy value, if it was able to skip a value (i.e. we had a value we'd otherwise generate in .pull-one, but we skipped it), or falsy value if there weren't any values to skip.

Now, the .skip method called on our Seq uses our new .skip-one method to cheaply skip through 2 items and then uses .pull-one to generate the third hash. Look at the timing now: 2.2s; the time it takes to generate a single hash.

However, we can kick it up a notch. While we won't notice a difference with our 3-item Seq, that user who was attempting to skip 1,000,000 items won't get the 2.2s time to generate the 1,000,000^th hash. They would also have to wait for 1,000,000 calls to .skip-one and @!stuff.shift. To optimize skipping over a bunch of items, we can implement the .skip-at-least method (for brevity, just our Iterator class is shown):

class :: does Iterator {
    has @.stuff;
    method pull-one {
        @!stuff
            ?? bcrypt-hash( @!stuff.shift, :15rounds )
            !! IterationEnd
    }
    method skip-one {
        return False unless @!stuff;
        @!stuff.shift;
        True
    }
    method skip-at-least (Int \n) {
        n == @!stuff.splice: 0, n
    }
}

The .skip-at-least method takes an Int of items to skip. It should skip as many as it can, and return a truthy value if it was able to skip that many items, and falsy value if the number of skipped items was fewer. Now, the user who skips 1,000,000 items will only have to suffer through a single .splice call.

For the sake of completeness, there's another skipping method defined by Iterator: .skip-at-least-pull-one. It follows the same semantics as .skip-at-least, except with .pull-one semantics for return values. Its default implemention involves just calling those two methods, short-circuiting and returning IterationEnd if the .skip-at-least returned a falsy value, and that default implementation is very likely good enough for all Iterators. The method exists as a convenience for Iterator users who call methods on Iterators and (at the moment) it's not used in core Rakudo Perl 6 by any methods that can be called on users' Seqs.

A so, so count...

There are two more optimization methods—.bool-only and .count-only—that do not have a default implementation. The first one returns True or False, depending on whether there are still items that can be generated by the Iterator (True if yes). The second one returns the number of items the Iterator can still produce. Importantly these methods must be able to do that without exhausting the Iterator. In other words, after finding these methods implemented, the user of our Iterator can call them and afterwards should still be able to .pull-one all of the items, as if the methods were never called.

Let's make an Iterator that will take an Iterable and .rotate it once per iteration of our Iterator until its tail becomes its head. Basically, we want this:

.say for rotator 1, 2, 3, 4;

# OUTPUT:
# [2 3 4 1]
# [3 4 1 2]
# [4 1 2 3]

This iterator will serve our purpose to study the two Iterator methods. For a less "made-up" example, try to find implementations of iterators for combinations and permutations routines in Perl 6 compiler's source code.

Here's a sub that creates our Seq with our shiny Iterator along with some code that operates on it and some timings for different stages of the program:

sub rotator (*@stuff) {
    Seq.new: class :: does Iterator {
        has int $!n;
        has int $!steps = 1;
        has     @.stuff is required;

        submethod TWEAK { $!n = @!stuff − 1 }

        method pull-one {
            if $!n-- > 0 {
                LEAVE $!steps = 1;
                [@!stuff .= rotate: $!steps]
            }
            else {
                IterationEnd
            }
        }
        method skip-one {
            $!n > 0 or return False;
            $!n--; $!steps++;
            True
        }
        method skip-at-least (Int \n) {
            if $!n > all 0, n {
                $!steps += n;
                $!n     −= n;
                True
            }
            else {
                $!n = 0;
                False
            }
        }
    }.new: stuff => [@stuff]
}

my $rotations := rotator ^5000;

if $rotations {
    say "Time after getting Bool: {now - INIT now}";

    say "We got $rotations.elems() rotations!";
    say "Time after getting count: {now - INIT now}";

    say "Fetching last one...";
    say "Last one's first 5 elements are: $rotations.tail.head(5)";
    say "Time after getting last elem: {now - INIT now}";
}

# OUTPUT:
# Time after getting Bool: 0.0230339
# We got 4999 rotations!
# Time after getting count: 26.04481484
# Fetching last one...
# Last one's first 5 elements are: 4999 0 1 2 3
# Time after getting last elem: 26.0466234

First things first, let's take a look at what we're doing in our Iterator. We take an Iterable (in the sub call on line 37, we use a Range object out of which we can milk 5000 elements in this case), shallow-clone it (using [ ... ] operator) and keep that clone in @!stuff attribute of our Iterator. During object instantiation, we also save how many items @!stuff has in it into $!n attribute, inside the TWEAK submethod.

For each .pull-one of the Iterator, we .rotate our @!stuff attribute, storing the rotated result back in it, as well as making a shallow clone of it, which is what we return for the iteration.

We also already implemented the .skip-one and .skip-at-least optimization methods, where we use a private $!steps attribute to alter how many steps the next .pull-one will .rotate our @!stuff by. Whenever .pull-one is called, we simply reset $!steps to its default value of 1 using the LEAVE phaser.

Let's check out how this thing performs! We store our precious Seq in $rotations variable that we first check for truthiness, to see if it has any elements in it at all; then we tell the world how many rotations we can fish out of that Seq; lastly, we fetch the last element of the Seq and (for screen space reasons) print the first 5 elements of the last rotation.

All three steps—check .Bool, check .elems, and fetch last item with .tail are timed, and the results aren't that pretty. While .Bool took relatively quick to complete, the .elems call took ages (26s)! That's actually not all of the damage. Recall from PART I of this series that both .Bool and .elems cache the Seq unless special methods are implemented in the Iterator. This means that each of those rotations we made are still there in memory, using up space for nothing! What are we to do? Let's try implementing those special methods .Bool and .elems are looking for!

This only thing we need to change is to add two extra methods to our Iterator that determinine how many elements we can generate (.count-only) and whether we have any elements to generate (.bool-only):

method count-only { $!n     }
method bool-only  { $!n > 0 }

For the sake of completeness, here is our previous example, with these two methods added to our Iterator:

sub rotator (*@stuff) {
    Seq.new: class :: does Iterator {
        has int $!n;
        has int $!steps = 1;
        has     @.stuff is required;

        submethod TWEAK { $!n = @!stuff − 1 }

        method count-only { $!n     }
        method bool-only  { $!n > 0 }

        method pull-one {
            if $!n-- > 0 {
                LEAVE $!steps = 1;
                [@!stuff .= rotate: $!steps]
            }
            else {
                IterationEnd
            }
        }
        method skip-one {
            $!n > 0 or return False;
            $!n--; $!steps++;
            True
        }
        method skip-at-least (\n) {
            if $!n > all 0, n {
                $!steps += n;
                $!n     −= n;
                True
            }
            else {
                $!n = 0;
                False
            }
        }
    }.new: stuff => [@stuff]
}

my $rotations := rotator ^5000;

if $rotations {
    say "Time after getting Bool: {now - INIT now}";

    say "We got $rotations.elems() rotations!";
    say "Time after getting count: {now - INIT now}";

    say "Fetching last one...";
    say "Last one's first 5 elements are: $rotations.tail.head(5)";
    say "Time after getting last elem: {now - INIT now}";
}

# OUTPUT:
# Time after getting Bool: 0.0087576
# We got 4999 rotations!
# Time after getting count: 0.00993624
# Fetching last one...
# Last one's first 5 elements are: 4999 0 1 2 3
# Time after getting last elem: 0.0149863

The code is nearly identical, but look at those sweet, sweet timings! Our entire program runs about 1,733 times faster because our Seq can figure out if and how many elements it has without having to iterate or rotate anything. The .tail call sees our optimization (side note: that's actually very recent) and it too doesn't have to iterate over anything and can just use our .skip-at-least optimization to skip to the end. And last but not least, our Seq is no longer being cached, so the only things kept around in memory are the things we care about. It's a huge win-win-win for very little extra code.

But wait... there's more!

Push it real good...

The Seqs we looked at so far did heavy work: each generated value took a relatively long time to generate. However, Seqs are quite versatile and at times you'll find that generation of a value is cheaper than calling .pull-one and storing that value somewhere. For cases like that, there're a few more methods we can implement to make our Seq perform better.

For the next example, we'll stick with the basics. Our Iterator will generate a sequence of positive even numbers up to the wanted limit. Here's what the call to the sub that makes our Seq looks like:

say evens-up-to 20; # OUTPUT: (2 4 6 8 10 12 14 16 18)

And here's the all of the code for it. The particular operation we'll be doing is storing all the values in an Array, by assigning to it:

sub evens-up-to {
    Seq.new: class :: does Iterator {
        has int $!n = 0;
        has int $.limit is required;
        method pull-one { ($!n += 2) < $!limit ?? $!n !! IterationEnd }
    }.new: :$^limit
}

my @a = evens-up-to 1_700_000;

say now - INIT now; # OUTPUT: 1.00765440

For a limit of 1.7 million, the code takes around a second to run. However, all we do in our Iterator is add some numbers together, so a lot of the time is likely lost in .pull-oneing the values and adding them to the Array, one by one.

In cases like this, implementing a custom .push-all method in our Iterator can help. The method receives one argument that is a reification target. We're pretty close to bare "metal" now, so we can't do anything fancy with the reification target object other than call .push method on it with a single value to add to the target. The .push-all always returns IterationEnd, since it exhausts the Iterator, so we'll just pop that value right into the return value of the method's Signature:

sub evens-up-to {
    Seq.new: class :: does Iterator {
        has int $!n = 0;
        has int $.limit is required;
        method pull-one {
            ($!n += 2) < $!limit ?? $!n !! IterationEnd
        }
        method push-all (\target --> IterationEnd) {
            target.push: $!n while ($!n += 2) < $!limit;
        }
    }.new: :$^limit
}

my @a = evens-up-to 1_700_000;
say now - INIT now; # OUTPUT: 0.91364949

Our program is now 10% faster; not a lot. However, since we're doing all the work in .push-all now, we no longer need to deal with state inside the method's body, so we can shave off a bit of time by using lexical variables instead of accessing object's attributes all the time. We'll make them use native int types for even more speed. Also, (at least currently), the += meta operator is more expensive than a simple assignment and a regular +; since we're trying to squeeze every last bit of juice here, let's take advantage of that as well. So what we have now is this:

sub evens-up-to {
    Seq.new: class :: does Iterator {
        has int $!n = 0;
        has int $.limit is required;
        method pull-one {
            ($!n += 2) < $!limit ?? $!n !! IterationEnd
        }
        method push-all (\target --> IterationEnd) {
            my int $limit = $!limit;
            my int $n     = $!n;
            target.push: $n while ($n = $n + 2) < $limit;
            $!n = $n;
        }
    }.new: :$^limit
}

my @a = evens-up-to 1_700_000;
say now - INIT now; # OUTPUT: 0.6688109

There we go. Now our program is 1.5 times faster than the original, thanks to .push-all. The gain isn't as dramatic as we what saw with other methods, but can come in quite handy when you need it.

There are a few more .push-* methods you can implement to, for example, do something special when your Seq is used in codes like...

for $your-seq -> $a, $b, $c { ... }

...where the Iterator would be asked to .push-exactly three items. The idea behind them is similar to .push-all: you push stuff onto the reification target. Their utility and performance gains are ever smaller, useful only in particular situations, so I won't be covering them.

It's worth noting the .push-all can be used only with Iterators that are not lazy, since... well... it expects you to push all the items. And what exactly are lazy Iterators? I'm so glad you asked!

A quick brown fox jumped over the lazy Seq

Let's pare down our previous Seq that generates even numbers down to the basics. Let's make it generate an infinite list of even numbers, using an anonymous state variable:

sub evens {
    Seq.new: class :: does Iterator {
        method pull-one { $ += 2 }
    }.new
}

put evens

Since the list is infinite, it'd take us an infinite time to fetch them all. So what exactly happens when we run the code above? It... quite predictably hangs when the put routine is called; it sits and patiently waits for our infinite Seq to complete. The same issue occurs when trying to assign our seq to a @-sigiled variable:

my @evens = evens # hangs

Or even when trying to pass our Seq to a sub with a slurpy parameter_Parameters):

sub meows (*@evens) { say 'Got some evens!' }
meows evens # hangs

That's quite an annoying problem. Fortunately, there's a very easy solution for it. But first, a minor detour to the land of naming clarification!

A rose by any other name would laze as sweet

In Perl 6 some things are or can be made "lazy". While it evokes the concept of on-demand or "lazy" evaluation, which is ubiquitous in Perl 6, things that are lazy in Perl 6 aren't just about that. If something is-lazy, it means it always wants to be evaluated lazily, fetching only as many items as needed, even in "mostly lazy" Perl 6 constructs that would otherwise eagerly consume even from sources that do on-demand generation.

For example, a sequence of lines read from a file would want to be lazy, as reading them all in at once has the potential to use up all the RAM. An infinite sequence would also want to be is-lazy because an eager evaluation would cause it to hang, as the sequence never completes.

So a thing that is-lazy in Perl 6 can be thought of as being infinite. Sometimes it actually will be infinite, but even if it isn't, it being lazy means it has similar consequences if used eagerly (too much CPU time used, too much RAM, etc).

Now back to our infinite list of even numbers. It sounds like all we have to do is make our Seq lazy and we do that by implementing .is-lazy method on our Iterator that simply returns True:

sub evens {
    Seq.new: class :: does Iterator {
        method pull-one { $ += 2 }
        method is-lazy (--> True) {}
    }.new
}

sub meows (*@evens) { say 'Got some evens!' }

put         evens; # OUTPUT: ...
my @evens = evens; # doesn't hang
meows       evens; # OUTPUT: Got some evens!

The put routine now detects its dealing with something terribly long and just outputs some dots. Assignment to Array no longer hangs (and will instead reify on demand). And the call to a slurpy doesn't hang either and will also reify on demand.

There's one more Iterator optimization method left that we should discuss...

A Sinking Ship

Perl 6 has sink context, similar to "void" context in other languages, which means a value is being discarded:

42;

# OUTPUT:
# WARNINGS for ...:
# Useless use of constant integer 42 in sink context (line 1)

The constant 42 in the above program is in sink context—its value isn't used by anything—and since it's nearly pointless to have it like that, the compiler warns about it.

Not all sinkage is bad however and sometimes you may find that gorgeous Seq on which you worked so hard is ruthlessly being sunk by the user! Let's take a look at what happens when we sink one of our previous examples, the Seq that generates up to limit even numbers:

sub evens-up-to {
    Seq.new: class :: does Iterator {
        has int $!n = 0;
        has int $.limit is required;
        method pull-one {
            ($!n += 2) < $!limit ?? $!n !! IterationEnd
        }
    }.new: :$^limit
}

evens-up-to 5_000_000; # sink our Seq

say now - INIT now; # OUTPUT: 5.87409072

Ouch! Iterating our Seq has no side-effects outside of the Iterator that it uses, which means it took the program almost six seconds to do absolutely nothing.

We can remedy the situation by implementing our own .sink-all method. Its default implementation .pull-ones until the end of the Seq (since Seqs may have useful side effects), which is not what we want for our Seq. So let's implement a .sink-all that does nothing!

sub evens-up-to {
    Seq.new: class :: does Iterator {
        has int $!n = 0;
        has int $.limit is required;
        method pull-one {
            ($!n += 2) < $!limit ?? $!n !! IterationEnd
        }
        method sink-all(--> IterationEnd) {}
    }.new: :$^limit
}

evens-up-to 5_000_000; # sink our Seq

say now - INIT now; # OUTPUT: 0.0038638

We added a single line of code and made our program 1,520 times faster—the perfect speed up for a program that does nothing!

However, doing nothing is not the only thing .sink-all is good for. Use it for clean up that would usually be done at the end of iteration (e.g. closing a file handle the Iterator was using). Or simply set the state of the system to what it would be at the end of the iteration (e.g. .seek a file handle to the end, for sunk Seq that produces lines from it). Or, as an alternative idea, how about warning the user their code might contain an error:

sub evens-up-to {
    Seq.new: class :: does Iterator {
        has int $!n = 0;
        has int $.limit is required;
        method pull-one {
            ($!n += 2) < $!limit ?? $!n !! IterationEnd
        }
        method sink-all(--> IterationEnd) {
            warn "Oh noes! Looks like you sunk all the evens!\n"
                ~ 'Why did you make them in the first place?'
        }
    }.new: :$^limit
}

evens-up-to 5_000_000; # sink our Seq

# OUTPUT:
# Oh noes! Looks like you sunk all the evens!
# Why did you make them in the first place?
# ...

That concludes our discussion on optimizing your Iterators. Now, let's talk about using Iterators others have made.

It's a marathon, not a sprint

With all the juicy knowledge about Iterators and Seqs we now possess, we can probably see how this piece of code manages to work without hanging, despite being given an infinite Range of numbers:

.say for ^∞ .grep(*.is-prime).map(* ~ ' is a prime number').head: 5;

# OUTPUT:
# 2 is a prime number
# 3 is a prime number
# 5 is a prime number
# 7 is a prime number
# 11 is a prime number

The infinite Range probably is-lazy. That .grep probably .pull-ones until it finds a prime number. The .map .pull-ones each of the .grep's values and modifies them, and .head allows at most 5 values to be .pull-oned from it.

In short what we have here is a pipeline of Seqs and Iterators where the Iterator of the next Seq is based on the Iterator of the previous one. For our study purposes, let's cook up a Seq of our own that combines all of the steps above:

sub first-five-primes (*@numbers) {
    Seq.new: class :: does Iterator {
        has     $.iter;
        has int $!produced = 0;
        method pull-one {
            $!produced++ == 5 and return IterationEnd;
            loop {
                my $value := $!iter.pull-one;
                return IterationEnd if $value =:= IterationEnd;
                return "$value is a prime number" if $value.is-prime;
            }
        }
    }.new: iter => @numbers.iterator
}

.say for first-five-primes ^∞;

# OUTPUT:
# 2 is a prime number
# 3 is a prime number
# 5 is a prime number
# 7 is a prime number
# 11 is a prime number

Our sub slurps up_Parameters) its positional arguments and then calls .iterator method on the @numbers Iterable. This method is available on all Perl 6 objects and will let us interface with the object using Iterator methods directly.

We save the @numbers's Iterator in one of the attributes of our Iterator as well as create another attribute to keep track of how many items we produced. In the .pull-one method, we first check whether we already produced the 5 items we need to produce, and if not, we drop into a loop that calls .pull-one on the other Iterator, the one we got from @numbers Array.

We recently learned that if the Iterator does not have any more values for us, it will return the IterationEnd constant. A constant whose job is to signal the end of iteration is finicky to deal with, as you can imagine. To detect it, we need to ensure we use the binding (:=), not the assignment (=) operator, when storing the value we get from .pull-one in a variable. This is because pretty much only the container identity (=:=) operator will accept such a monstrosity, so we can't stuff the value we .pull-one into just any container we please.

In our example program, if we do find that we received IterationEnd from the source Iterator, we simply return it to indicate we're done. If not, we repeat the process until we find a prime number, which we then put into our desired string and that's what we return from our .pull-one.

All the rest of the Iterator methods we've learned about can be called on the source Iterator in a similar fashion as we called .pull-one in our example.

Conclusion

Today, we've learned a whole ton of stuff! We now know that Seqs are powered by Iterator objects and we can make custom iterators that generate any variety of values we can dream about.

The most basic Iterator has only .pull-one method that generates a single value and returns IterationEnd when it has no more values to produce. It's not permitted to call .pull-one again, once it generated IterationEnd and we can write our .pull-one methods with the expectation that will never happen.

There are plenty of optimization opportunities a custom Iterator can take advantage of. If it can cheaply skip through items, it can implement .skip-one or .skip-at-least methods. If it can know how many items it'll produce, it can implement .bool-only and .count-only methods that can avoid a ton of work and memory use when only certain values of a Seq are needed. And for squeezing the very last bit of performance, you can take advantage of .push-all and other .push-* methods that let you push values onto the target directly.

When your Iterator .is-lazy, things will treat it with extra care and won't try to fetch all of the items at once. And we can use the .sink-all method to avoid work or warn the user of potential mistakes in their code, when our Seq is being sunk.

Lastly, since we know how to make Iterators and what their methods do, we can make use of Iterators coming from other sources and call methods on them directly, manipulating them just how we want to.

We now have all the tools to work with Seq objects in Perl 6. In the PART III of this series, we'll learn how to compactify all of that knowledge and skillfully build Seqs with just a line or two of code, using the sequence operator.

Stay tuned!

-Ofun

Read this article on Perl6.Party

I vividly recall my first steps in Perl 6 were just a couple of months before the first stable release of the language in December 2015. Around that time, Larry Wall was making a presentation and showed a neat feature—the sequence operator—and it got me amazed about just how powerful the language is:

# First 12 even numbers:
say (2, 4 … ∞)[^12];      # OUTPUT: (2 4 6 8 10 12 14 16 18 20 22 24)

# First 10 powers of 2:
say (2, 2², 2³ … ∞)[^10]; # OUTPUT: (2 4 8 16 32 64 128 256 512 1024)

# First 13 Fibonacci numbers:
say (1, 1, *+* … ∞)[^13]; # OUTPUT: (1 1 2 3 5 8 13 21 34 55 89 144 233)

The ellipsis (…) is the sequence operator and the stuff it makes is the Seq object. And now, a year and a half after Perl 6's first release, I hope to pass on my amazement to a new batch of future Perl 6 programmers.

This is a 3-part series. In PART I of this article we'll talk about what Seq s are and how to make them without the sequence operator. In PART II, we'll look at the thing-behind-the-curtain of Seq's: the Iterator type and how to make Seqs from our own Iterators. Lastly, in PART III, we'll examine the sequence operator in all of its glory.

Note: I will be using all sorts of fancy Unicode operators and symbols in this article. If you don't like them, consult with the Texas Equivalents page for the equivalent ASCII-only way to type those elements.

PART I: What the Seq is all this about?

The Seq stands for Sequence and the Seq object provides a one-shot way to iterate over a sequence of stuff. New values can be generated on demand—in fact, it's perfectly possible to create infinite sequences—and already-generated values are discarded, never to be seen again, although, there's a way to cache them, as we'll see.

Sequences are driven by Iterator objects that are responsible for generating values. However, in many cases you don't have to create Iterators directly or use their methods while iterating a Seq. There are several ways to make a Seq and in this section, we'll talk about gather/take construct.

I gather you'll take us to...

The gather statement and take routine are similar to "generators" and "yield" statement in some other languages:

my $seq-full-of-sunshine := gather {
    say  'And nobody cries';
    say  'there’s only butterflies';

    take 'me away';
    say  'A secret place';
    say  'A sweet escape';

    take 'meee awaaay';
    say  'To better days'    ;

    take 'MEEE AWAAAAYYYY';
    say  'A hiding place';
}

Above, we have a code block with lines of song lyrics, some of which we say (print to the screen) and others we take (to be gathered). Just like, .say can be used as either a method or a subroutine, so you can use .take as a method or subroutine, there's no real difference; merely convenience.

Now, let's iterate over $seq-full-of-sunshine and watch the output:

for $seq-full-of-sunshine {
    ENTER say '▬▬▶ Entering';
    LEAVE say '◀▬▬ Leaving';

    say "❚❚ $_";
}

# OUTPUT:
# And nobody cries
# there’s only butterflies
# ▬▬▶ Entering
# ❚❚ me away
# ◀▬▬ Leaving
# A secret place
# A sweet escape
# ▬▬▶ Entering
# ❚❚ meee awaaay
# ◀▬▬ Leaving
# To better days
# ▬▬▶ Entering
# ❚❚ MEEE AWAAAAYYYY
# ◀▬▬ Leaving
# A hiding place

Notice how the say statements we had inside the gather statement didn't actualy get executed until we needed to iterate over a value that take routines took after those particular say lines. The block got stopped and then continued only when more values from the Seq were requested. The last say call didn't have any more takes after it, and it got executed when the iterator was asked for more values after the last take.

That's exceptional!

The take routine works by throwing a CX::Take control exception that will percolate up the call stack until something takes care of it. This means you can feed a gather not just from an immediate block, but from a bunch of different sources, such as routine calls:

multi what's-that (42)                     { take 'The Answer'            }
multi what's-that (Int $ where *.is-prime) { take 'Tis a prime!'          }
multi what's-that (Numeric)                { take 'Some kind of a number' }

multi what's-that   { how-good-is $^it                   }
sub how-good-is ($) { take rand > ½ ?? 'Tis OK' !! 'Eww' }

my $seq := gather map &what's-that, 1, 31337, 42, 'meows';

.say for $seq;

# OUTPUT:
# Some kind of a number
# Tis a prime!
# The Answer
# Eww

Once again, we iterated over our new Seq with a for loop, and you can see that take called from different multies and even nested sub calls still delivered the value to our gather successfully:

The only limitation is you can't gather takes done in another Promise or in code manually cued in the scheduler:

gather await start take 42;
# OUTPUT:
# Tried to get the result of a broken Promise
#   in block <unit> at test.p6 line 2
#
# Original exception:
#     take without gather

gather $*SCHEDULER.cue: { take 42 }
await Promise.in: 2;
# OUTPUT: Unhandled exception: take without gather

However, nothing's stopping you from using a Channel to proxy your data to be taken in a react block.

my Channel $chan .= new;
my $promise = start gather react whenever $chan { .take }

say "Sending stuff to Channel to gather...";
await start {
    $chan.send: $_ for <a b c>;
    $chan.close;
}
dd await $promise;

# OUTPUT:
# Sending stuff to Channel to gather...
# ("a", "b", "c").Seq

Or gathering takes from within a Supply:

my $supply = supply {
    take 42;
    emit 'Took 42!';
}

my $x := gather react whenever $supply { .say }
say $x;

# OUTPUT: Took 42!
# (42)

Stash into the cache

I mentioned earlier that Seqs are one-shot Iterables that can be iterated only once. So what exactly happens when we try to iterate them the second time?

my $seq := gather take 42;
.say for $seq;
.say for $seq;

# OUTPUT:
# 42
# This Seq has already been iterated, and its values consumed
# (you might solve this by adding .cache on usages of the Seq, or
# by assigning the Seq into an array)

A X::Seq::Consumed exception gets thrown. In fact, Seqs do not even do the Positional role, which is why we didn't use the @ sigil that type- checks for Positional on the variables we stored Seqs in.

The Seq is deemed consumed whenever something asks it for its Iterator after another thing grabbed it, like the for loop would. For example, even if in the first for loop above we would've iterated over just 1 item, we wouldn't be able to resume taking more items in the next for loop, as it'd try to ask for the Seq's iterator that was already taken by the first for loop.

As you can imagine, having Seqs always be one-shot would be somewhat of a pain in the butt. A lot of times you can afford to keep the entire sequence around, which is the price for being able to access its values more than once, and that's precisely what the Seq.cachemethod does:

my $seq := gather { take 42; take 70 };
$seq.cache;

.say for $seq;
.say for $seq;

# OUTPUT:
# 42
# 70
# 42
# 70

As long as you call .cache before you fetch the first item of the Seq, you're good to go iterating over it until the heat death of the Universe (or until its cache noms all of your RAM). However, often you do not even need to call .cache yourself.

Many methods will automatically .cache the Seq for you:

• .Str, .Stringy, .fmt, .gist, .perl methods always .cache
• .AT-POS and .EXISTS-POS methods, or in other words, Positional indexing like $seq[^10], always .cache
• .elems, .Numeric, and .Int will .cache the Seq, unless the underlying Iterator provides a .count-only method (we'll get to those in PART II)
• .Bool will .cache unless the underlying Iterator provides .bool-only or .count-only methods

There's one more nicety with Seqs losing their one-shotness that you may see refered to as PositionalBindFailover. It's a role that indicates to the parameter binder that the type can still be converted into a Positional, even when it doesn't do Positional role. In plain English, it means you can do this:

sub foo (@pos) { say @pos[1, 3, 5] }

my $seq := 2, 4 … ∞;
foo $seq; # OUTPUT: (4 8 12)

We have a sub that expects a Positional argument and we give it a Seq which isn't Positional, yet it all works out, because the binder .caches our Seq and uses the List the .cache method returns to be the Positional to be used, thanks to it doing the PositionalBindFailover role.

Last, but not least, if you don't care about all of your Seq's values being generated and cached right there and then, you can simply assign it to a @ sigiled variable, which will reify the Seq and store it as an Array:

my @stuff = gather {
    take 42;
    say "meow";
    take 70;
}

say "Starting to iterate:";
.say for @stuff;

# OUTPUT:
# meow
# Starting to iterate:
# 42
# 70

From the output, we can see say "meow" was executed on assignment to @stuff and not when we actually iterated over the value in the for loop.

Conclusion

In Perl 6, Seqs are one-shot Iterables that don't keep their values around, which makes them very useful for iterating over huge, or even infinite, sequences. However, it's perfectly possible to cache Seq values and re-use them, if that is needed. In fact, many of the Seq's methods will automatically cache the Seq for you.

There are several ways to create Seqs, one of which is to use the gather and take where a gather block will stop its execution and continue it only when more values are needed.

In parts II and III, we'll look at other, more exciting, ways of creating Seqs. Stay tuned!

-Ofun

Read this article on Perl6.Party

As some recall, Rakudo's 2017.04 release was somewhat of a trainwreck. It was clear the quality assurance of releases needed to be kicked up a notch. So today, I'll talk about what progress we've made in that area.

Define The Problem

A particular problem that plagued the 2017.04 release were big changes and refactors made in the compiler that passed all the 150,000+ stresstests, however still caused issues in some ecosystem modules and users' code.

The upcoming 2017.06 has many, many more big changes:

• IO::ArgFiles were entirely replaced with the new IO::CatHandle implementation
• IO::Socket got a refactor and sync sockets no longer use libuv
• IO::Handle got a refactor with encoding and sync IO no longer uses libuv
• Sets/Bags/Mixes got optimization polish and op semantics finalizations
• Proc was refactored to be in terms of Proc::Async

The IO and Proc stuff is especially impactful, as it affects precomp and module loading as well. Merely passing stresstests just wouldn't give me enough of peace of mind of a solid release. It was time to extend the testing.

Going All In

The good news is I didn't actually have to write any new tests. With 836 modules in the Perl 6 ecosystem, the tests were already there for the taking. Best of all, they were mostly written without bias due to implementation knowledge of core code, as well as have personal style variations from hundreds of different coders. This is all perfect for testing for any regressions of core code. The only problem is running all that.

While there's a budding effort to get CPANTesters to smoke Perl 6 dists, it's not quite the data I need. I need to smoke a whole ton of modules on a particular pre-release commit, while also smoking them on a previous release on the same box, eliminating setup issues that might contribute to failures, as well as ensuring the results were for the same versions of modules.

My first crude attempt involved firing up a 32-core Google Compute Engine VM and writing a 60-line script that launched 836 Proc::Asyncs—one for each module.

Other than chewing through 125 GB of RAM with a single Perl 6 program, the experiment didn't yield any useful data. Each module had to wait for locks, before being installed, and all the Procs were asking zef to install to the same location, so dependency handling was iffy. I needed a more refined solution...

Procs, Kernels, and Murder

So, I started to polish my code. First, I wrote Proc::Q module that let me queue up a bunch of Procs, and scale the number of them running at the same time, based on the number of cores the box had. Supply.throttle core feature made the job a piece of cake.

However, some modules are naughty or broken and I needed a way to kill Procs that take too long to run. Alas, I discovered that Proc::Async.kill had a bug in it, where trying to simultaneously kill a bunch of Procs was failing. After some digging I found out the cause was $*KERNEL.signal method the .kill was using isn't actually thread safe and the bug was due to a data race in initialization of the signal table.

After refactoring Kernel.signal, and fixing Proc::Async.kill, I released Proc::Q module—my first module to require (at the time) the bleedest of bleeding edges: a HEAD commit.

Going Atomic

After cooking up boilerplate DB and Proc::Q code, I was ready to toast the ecosystem. However, it appeared zef wasn't designed, or at least well-tested, in scenarious where up to 40 instances were running module installations simultaneously. I was getting JSON errors from reading ecosystem JSON, broken cache files (due to lack of file locking), and false positives in installations because modules claimed they were already installed.

I initially attempted to solve the JSON errors by looking at an Issue in the ecosystem repo about the updater script not writing atomically. However, even after fixing the updater script, I was still getting invalid JSON errors from zef when reading ecosystem data.

It might be due to something in zef, but instead of investigating it further, I followed ugexe++'s advice and told zef not to fetch ecosystem in each Proc. The broken cache issues were similarly eliminated by disabling caching support. And the false positives were eliminated telling each zef instance to install the tested module into a separate location.

The final solution involved programatically editing zef's config file before a toast run to disable auto-updates of CPAN and p6c ecosystem data, and then in individual Procs zef module install command ended up being:

«zef --/cached --debug install "$module" "--install-to=inst#$where"»

Where $where is a per-module, per-rakudo-commit location. The final issue was floppy test runs, which I resolved by re-testing failed modules one more time, to see if the new run succeeds.

Time is Everything

The toasting of the entire ecosystem on HEAD and 2017.05 releases took about three hours on a 24-core VM, while being unattended. While watching over it and killing the few hanging modules at the end without waiting for them to time out makes a single-commit run take about 65 minutes.

I also did a toast run on a 64-core VM...

Overall, the run took me 50 minutes, and I had to manually kill some modules' tests. However, looking at CPU utilization charts, it seems the run sat idle for dozens of minutes before I came along to kill stuff:

So I think after some polish of avoiding hanging modules and figuring out why (apparently) Proc::Async.kill still doesn't kill everything, the runs can be entirely automated and a single run can be completed in about 20-30 minutes.

This means that even with last-minute big changes pushed to Rakudo, I can still toast the entire ecosystem reasonably fast, detect any potential regressions, fix them, and re-test again.

Reeling In The Catch

The Toaster database is available for viewing at toast.perl6.party. As more commits get toasted, they get added to the database. I plan to clear them out after each release.

The toasting runs I did so far weren't just a chance to play with powerful hardware. The very first issue was detected when toasting Clifford module.

The issue was to do with Lists of Pairs with same keys coerced into a MixHash, when the final accumulative weight was zero. The issue was introduced on June 7th and it took me about an hour of digging through the module's guts to find it. Considering it's quite an edge case, I imagine without the toaster runs it would take a lot longer to identify this bug. lizmat++ squashed this bug hours after identification and it never made it into any releases.

The other issue detected by toasting had to do with the VM-backed decoder serialization introduced during IO refactor and jnthn++ fixed it a day after detection. One more bug had to do with Proc refactor making Proc not synchronous-enough. It was mercilessly squashed, while fixing a couple of longstanding issues with Proc.

All of these issues weren't detected by the 150,000+ tests in the testsuite and while an argument can be made that the tests are sparse in places, there's no doubt the Toaster has paid off for the effort in making it by catching bugs that might've otherwise made it into the release.

The Future

The future plans for the Toaster would be first to make it toast on more platforms, like Windows and MacOS. Eventually, I hope to make toast runs continuous, on less-powerful VMs that are entirely automated. An IRC bot would watch for any failures and report them to the dev channel.

Conclusion

The ecosystem Toaster lets core devs test a Rakudo commit on hundreds of software pieces, made by hundreds of different developers, all within a single hour. During its short existence, the Toaster already found issues with ecosystem infrastructure, highly-multi-threaded Perl 6 programs, as well as detected regressions and new bugs that we were able to fix before the release.

The extra testing lets core devs deliver higher-quality releases, which makes Perl 6 more trustworthy to use in production-quality software. The future will see the Toaster improved to test on a wider range of systems, as well as being automated for continued extended testing.

And most importantly, the Toaster makes it possible for any Perl 6 programmer to help core development of Perl 6, by simply publishing a module.

-Ofun

This document is the May, 2017 progress report for TPF Standardization, Test Coverage, and Documentation of Perl 6 I/O Routines grant. I believe I reasonably satisfied the goals of the grant and consider it completed. This is the final report and may reference some of the work/commits previously mentioned in monthly reports.

Thank You!

I'd like to thank all the donors that support The Perl Foundation who made this grant possible. It was a wonderful learning experience for me, and it brings me joy to look back and see Perl 6 improved due to this grant.

Thank You!

Completeness Criteria

Here are the original completeness criteria (in bold) that are listed on the original grant proposal and my comments on their status:

• rakudo repository will contain the IO Action Plan document and it will be fully implemented. The promised document exists. It's fully implemented except for three items that I listed on the IO Action Plan, but which are currently a bit beyond my skill level to implement. I hope to do them eventually, but outside the scope of this grant. They are:
  • IO::Handle's Closed status. My original proposal would cause some perfomance issues, so it was decided to improve MoarVM errors instead.
  • Optimize multiple stat calls. This involves creating a new nqp op, with code for it implemented in MoarVM and JVM backends.
  • Use typed exceptions instead of X::AdHoc. I made typed exceptions be thrown whereever I could. The rest require VM-level exceptions and is on the same level as the handle closed status issue (first item above).
• All of the I/O routines will have tests in roast and documented on docs.perl6.org. If any of the currently implemented but unspecced routines are decided against being included in Perl 6 Language, their implementation will no longer be available in Rakudo. To the best of my knowledge, this is completed in full.
• The test coverage tool will report all I/O routines as covered and the information will be visible on perl6.wtf (Perl 6's Wonderful Test Files) website. Note: due to current experimental status of the coverage tool, its report may still show some lines or conditionals untested despite them actually being tested; however, it will show the lines where routines' names are specified as covered. To the best of my knowledge, all IO routines currently have tests covering them. Due to its experimental status, the coverage tool shows some attributes as uncovered. I did manually verify all the attributes/routines whose names the tool shows as uncovered contain tests for them. One exception is IO::Notification type (and IO::Path.watch method). While it has full coverage for OSX operating system, it lacks it for other OSes. I tried writing some tests for it, but it looks like the behaviour of the nqp op handling these is broken on Linux and the class needs more work.

Extra Deliverables

I produced these extra deliverables while working on the grant:

• The Definitive I/O Guide. Providing tutorial-like documentation for Perl 6's I/O, including documenting some of the bad practices I noticed in the ecosystem (and even a Perl 6 book!) and the correct way to perform those tasks. (N.B. as I write this report, the guide could still use a few extra sections to be considered "The Definitive"; I'll write them in upcoming weeks)
• Performance improvements. I made 23 performance-enhancing commits, with many commits making things more than 200% faster, with highest improvement making a routine 6300% faster.
• Trait::IO module. Provides does auto-close pseudo-trait to simplify closing of IO handles.
• IO::Path::ChildSecure module. Due to large ecosystem usage, IO::Path.child was left as is until 6.d language, at which point it will be made secure (as outlined in the IO Plan). This module provides the secure version in the mean time.
• IO::Dir module. Provides IO::Path.dir-like functionality, with ability to close open directory without needing to fully exhaust the returned Seq.
• Die module. Implements Perl-5-like behaviour for &die routine.
• The "Map of Perl 6 Routines" (or rather the "table") is available on map.perl6.party with its code in perl6/routine-map repo. In near future, I plan to use it to identify incorrect or incomplete entries in our documentation

In addition, I plan to complete these modules some time in the future; the ideas for them were birthed while working on the grant: - NL module. Targeted for use in one liners, the module will provide $*NL dynvar that behaves like Perl 5's $. variable (providing current $*ARGFILES's file's line number). Its implementation became possible thanks to newly-implemented IO::CatHandle type - FastIO module. A re-imagination of core IO, the biggest part of which will be the removal of (user-exposed) use of IO::Spec::* types and $*SPEC variable, which—it is believed—will provide improved performance over core IO. The module is a prototype for some of the proposals that were made during the IO grant and if it offers significant improvements over core IO, its ideas will be used by core IO in future language versions.

Work Performed in May

For the work done in May, many of my commits went into going through the IO routine list, and adding missing tests and documentation, along with fixing bugs (and reporting new ones I found).

The major work was implementation of the IO::CatHandle class that fixed all of the bugs and NYIs with the $*ARGFILES. This work saw the addition of 372 lines of code, 800 lines of tests and 793 lines of documentation.

Work by Other Core Members

jnthn++ completed the handle encoding refactor that will eventually let us get rid of using libuv for syncronous IO and, more importantly, allow us to support user-defined encoders/decoders.

Along with fixing a bunch of bugs, this work altered the performance landscape for IO operations (i.e. some operations may now be a bit faster, others a bit slower), though overall the performance appeared to stay the same.

Tickets Fixed

• RT#130456 $*HOME blows up if HOME isn't set
• RT#126935 bad .perl for paths with pipe characters (was already fixed; just added an extra test)
• RT#131185 IO::Path.perl doesn't roundtrip
• RT#130454 tmpdir tries to change the current working directory (routine was removed as part of IO work)
• RT#130455 Should I be able to change the temporary directory?
• RT#130715 IO::Handle::close shouldn't decide what's a failure (not a bug, but user's confusion; explained and rejected)
• RT#131242 IO::Path.move/.copy hangs/trashes file when target/source are same

Grant Commits

During this grant, I've made 417 commits, that are: 134 Rakudo commits + 23 performance-enchancing Rakudo commits + 114 Perl 6 Specification commits + 146 documentation commits,

Performance Rakudo Commits

I've made 23 performance enchancing commits to Rakudo's repository:

• 4032953 Make IO::Handle.open 75% faster
• dcf1bb2 Make IO::Spec::Unix.rel2abs 35% faster
• c13480c IO::Path.slurp: make 12%-35% faster; propagate Failures
• 0e36bb2 Make IO::Spec::Win32!canon-cat 2.3x faster
• c6fd736 Make IO::Spec::Win32.is-absolute about 63x faster
• 894ba82 Make IO::Spec::Win32.split about 82% faster
• 277b6e5 Make IO::Spec::Unix.rel2abs 2.9x faster
• 74680d4 Make IO::Path.is-absolute about 80% faster
• ff23416 Make IO::Path.is-relative about 2.1x faster
• d272667 Make IO::Spec::Unix.join about 40% faster
• 50429b1 Make IO::Handle.put($x) about 5%-35% faster
• 204ea59 Make &say(**@args) 70%− faster
• 6d7fc8e Make &put(**@args) up to 70% faster
• 76af536 Make 1-arg IO::Handle.say up to 2x faster
• aa72bde Remove dir's :absolute and :Str; make up to 23% faster
• 48cf0e6 Make IO::Spec::Cygwin.is-absolute 21x faster
• c96727a Fix combiners on SPEC::Win32.rel2abs; make 6% faster
• 0547979 Make IO::Spec::Unix.path consistent and 4.6x faster
• 8992af1 Fix IO::Spec::Win32.path and make 26x faster
• 7d6fa73 Make IO::Spec::Win32.catpath 47x faster
• 494659a Make IO::Spec::Win32.join 26x faster
• 6ca702f Make IO::Spec::Unix.splitdir 7.7x faster
• 2816ef7 Make IO::Spec::Win32.splitdir 25x faster

Non-Performance Rakudo Commits

Other than perf commits, I've also made 134 commits to the Rakudo's repository:

• dd4dfb1 Fix crash in IO::Special .WHICH/.Str
• 76f7187 Do not cache IO::Path.e results
• 212cc8a Remove IO::Path.Bridge
• a01d679 Remove IO::Path.pipe
• 55abc6d Improve IO::Path.child perf on *nix
• 4fdebc9 Make IO::Spec::Unix.split 36x Faster
• 0111f10 Make IO::Spec::Unix.catdir 3.9x Faster
• fa9aa47 Make R::I::SET_LINE_ENDING_ON_HANDLE 4.1x Faster
• c360ac2 Fix smartmatch of Cool ~~ IO::Path
• 0c7e4a0 Do not capture args in .IO method
• 9d8d7b2 Log all changes to plan made during review period
• 87987c2 Removerole IOand its .umask method
• 36ad92a Remove 15 methods from IO::Handle
• a5800a1 Implement IO::Handle.spurt
• aa62cd5 Remove &tmpdir and &homedir
• a0ef2ed Improve &chdir, &indir, and IO::Path.chdir
• ca1acb7 Fix race in &indir(IO::Path …)
• 2483d68 Fix regression in &chdir's failure mode
• 5464b82 Improve &*chdir
• 4c31903 Add S32-io/chdir-process.t to list of test files to run
• cb27bce Clean up &open and IO::Path.open
• 099512b Clean up and improve all spurt routines
• b62d1a7 Give $*TMPDIR a container
• b1e7a01 Implement IO::Path.extension 2.0
• 15a25da Fix ambiguity in empty extension vs no extension
• 50aea2b Restore IO::Handle.IO
• 966a7e3 Implement IO::Path.concat-with
• 94a6909 Clean up IO::Spec::Unix.abs2rel a bit
• a432b3d Remove IO::Path.abspath (part 2)
• 954e69e Fix return value of IO::Special methods
• 67f06b2 Run S32-io/io-special.t test file
• a0b82ed Make IO::Path::* actually instantiate a subclass
• 0c8bef5 Implement :parent in IO::Spec::Cygwin.canonpath
• 0a442ce Remove type constraint in IO::Spec::Cygwin.canonpath
• b4358af Delete code for IO::Spec::Win32.catfile
• e681498 Make IO::Path throw when path contains NUL byte
• 6a8d63d Implement :completely param in IO::Path.resolve
• b6838ee Remove .f check in .z
• 184d499 Make IO::Handle.Supply respect handle's mode
• f1b4af7 Implement IO::Handle.slurp
• 90da80f Rework read methods in IO::Path/IO::Handle
• 8c09c84 Fix symlink and link routines
• da1dea2 Fix &symlink and &link
• 7f73f92 Make IO::Path.new-from-absolute-path private
• ff97083 Straighten up rename, move, and copy
• 0d9ecae Remove multi-dir &mkdir
• 6ee71c2 Coerce mode in IO::Path.mkdir to Int
• d46e8df Add IO::Pipe .path and .IO methods
• c01ebea Make IO::Path.mkdir return invocant on success
• 1f689a9 Fix up IO::Handle.Str
• 490ffd1 Do not use self.Str in IO::Path errors
• 40217ed Swap .child to .concat-with in all the guts
• fd503f8 Revert "Removerole IOand its .umask method"
• c95c4a7 Make IO::Path/IO::Special do IO role
• 214198b Implement proper args for IO::Handle.lock
• 9a2446c Move Bool return value to signature
• 51e4629 Amend rules for last part in IO::Path.resolve
• b8458d3 Rewordmethod childfor cleaner code
• 1887114 Implement IO::Path.child-secure
• 9d8e391 Fix IO::Path.resolve with combiners; timotimo++
• 0b5a41b Rename IO::Path.concat-with to .add
• a98b285 Remove IO::Path.child-secure
• 8bacad8 Implement IO::Path.sibling
• 7112a08 Add :D on invocant for file tests
• b2a64a1 Fix $*CWD inside IO::Path.dir's :test Callable
• 6fa4bbc Straighten out &slurp/&spurt/&get/&getc/&close
• 34b58d1 Straighten out &lines/&words
• d0cd137 Make dir take any IO(), not just Cool
• 7412184 Make $*HOME default to Nil, not Any
• 475d9bc Fix display of backslashes in IO::Path.gist
• 6ef2abd Revert "Fix display of backslashes in IO::Path.gist"
• 134efd8 Fix .perl for IO::Path and subclasses
• 69320e7 Fix .IO on :U of IO::Path subclasses
• eb8d006 Make IO::Handle.iterator a private lines iterator
• 08a8075 Fix IO::Path.copy/move when source/target are same
• 973338a Fix IO::Handle.comb/.split; make them .slurp
• b43ed18 Make IO::Handle.flush fail with typed exceptions
• 276d4a7 Remove .tell info in IO::Handle.gist
• f4309de Fix IO::Spec::Unix.is-absolute for combiners on /
• 06d8800 Fix crash when setting .nl-in ...
• 7e9496d Make IO::Handle.encoding settable via .new
• 95e49dc Make IO::Handle.open respect attribute values
• 6ed14ef Remove:directoryfrom IO::Spec::*.split
• 9021a48 Make IO::Path.parts a Map instead of Hash
• a282b8c Fix IO::Handle.perl.EVAL roundtrippage
• a412788 Make IO::Path.resolve set CWD to $!SPEC.dir-sep
• 84502dc Implement $limit arg for IO::Handle.words
• 613bdcf Make IO::Handle.print/.put sig consistent
• 0646d3f Allow no-arg &prompt
• 4a8aa27 Implement IO::CatHandle.close
• 4ad8b17 Implement IO::CatHandle.get
• 3b668b6 Implement IO::CatHandle.getc
• 25b664a Implement IO::CatHandle.words
• 7ebc386 Implement IO::CatHandle.slurp
• 52b34b7 Implement IO::CatHandle.comb/.split
• beaa925 Implement IO::CatHandle.read
• ccc90fd Implement IO::CatHandle.readchars
• 40f4dc9 Implement IO::CatHandle.Supply
• 0c9aea7 Implement IO::CatHandle.encoding
• ee1e185 Implement IO::CatHandle.eof
• 80686a7 Implement IO::CatHandle.t/.path/.IO/.native-descriptor
• 993de50 Implement IO::CatHandle.gist/.Str/.opened/.open
• 677c4ea Implement IO::CatHandle.lock/.unlock/.seek/.tell
• e657ed1 Implement IO::CatHandle.chomp/.nl-in
• a452e42 Implement IO::CatHandle.on-switch
• f539a62 Swap IO::ArgFiles to IO::CatHandle impl
• fa7aa1c Implement IO::CatHandle.perl method
• 21fd2c4 Remove IO::Path.watch
• 65941b2 Revert "Remove IO::Path.watch"
• a47a78f Remove useless :SPEC/:CWD on some IO subs
• d13d9c2 Throw out IO::Path.int

Perl 6 Specification Commits

I've made 114 commits to the Perl 6 Specification (roast) repository:

• 63370fe Test IO::Special .WHICH/.Str do not crash
• 465795c Test IO::Path.lines(*) does not crash
• 091931a Expand &open tests
• 8d6ca7a Cover IO::Path.ACCEPTS
• 14b6844 Use Numeric instead of IO role in dispatch test
• 5a7a365 Expand IO::Spec::*.tmpdir tests
• f48198f Test &indir
• bd46836 Amend &indir race tests
• 04333b3 Test &indir fails with non-existent paths by default
• 73a5448 Remove two fudged &chdir tests
• 86f79ce Expand &chdir tests
• 430ab89 Test &*chdir
• 86c5f9c Delete qp{} tests
• 3c4e81b Test IO::Path.Str works as advertised
• ba3e7be Merge S32-io/path.t and S32-io/io-path.t
• 79ff022 Expand &spurt and IO::Path.spurt tests
• 1d4e881 Test $*TMPDIR can betemped
• b23e53e Test IO::Path.extension
• 2f09f18 Fix incorrect test
• 305f206 Test empty-string extensions in IO::Path.extension
• 0e47f25 Test IO::Path.concat-with
• e5dc376 Expand IO::Path.accessed tests
• 43ec543 Cover methods of IO::Special
• bd8d167 Test IO::Path::* instantiate a subclass
• d8707e7 Cover IO::Spec::Unix.basename
• c3c51ed Cover IO::Spec::Win32.basename
• 896033a Cover IO::Spec::QNX.canonpath
• 7c7fbb4 Cover :parent arg in IO::Spec::Cygwin.canonpath
• 8f73ad8 Change \0 roundtrip test to \t roundtrip test
• b16fbd3 Add tests to check nul byte is rejected
• ee7f05b Move is-path sub to top so it can be reused
• a809f0f Expand IO::Path.resolve tests
• feecaf0 Expand file tests
• a4c53b0 Use bin IO::Handle to test its .Supply
• 7e4a2ae Swap .slurp-rest to .slurp
• d4353b6 Rewrite .l on broken symlinks test
• 416b746 Test symlink routines
• 8fa49e1 Testlinkroutines
• 637500d Spec IO::Pipe.path/.IO returns IO::Path type object
• 64ff572 Cover IO::Path/IO::Pipe's .Str/.path/.IO
• 4194755 Test IO::Handle.lock/.unlock
• a716962 Amend rules for last part in IO::Path.resolve
• f3c5dae Test IO::Path.child-secure
• 92217f7 Test IO::Path.child-secure with combiners
• 39677c4 IO::Path.concat-with got renamed to .add
• 7a063b5 Fudge .child-secure tests
• 3b36d4d Test IO::Path.sibling
• 41b7f9f Test $*CWD in IO::Path.dir(:test) Callable
• 18d9c04 Cover IO::Handle.spurt
• 8f78ca6 Test &words with IO::ArgFiles
• ea137f6 Cover IO::Handle.tell
• 71a6423 Add $*HOME tests
• 95d68a2 Test IO::Path.gist does escapes of backslashes
• de89d25 Revert "Test IO::Path.gist does escapes of backslashes"
• 9e8b154 Test IO::Handle.close can be...
• 853f76f Test IO::Pipe.close returns pipe's Proc
• d543e75 Test IO::Handle.DESTROY closes the handle
• 1ed18b4 Add test for .perl.EVAL roundtrip with combiners
• 704210c Test we can roundtrip IO::Path.perl
• 2689eb1 Test .IO on :U of IO::Path subclasses
• 40353f1 Test for IO::Handle:D { ... } loops over handle
• 4fdb850 Test IO::Path.copy/move when source/target are same
• 98917dc Test IO::Path.dir's absoluteness behaviour
• 71eebc7 Test IO::Spec::Unix.extension
• 4495615 Test IO::Handle.flush
• 60f5a6d Test IO::Handle.t when handle is a TTY
• 31e3993 Test IO::Path*.gist
• c481433 Test .is-absolute method for / with combiners
• 8ee0a0a Test IO::Spec::Win32.rel2abs with combiners
• a41027f Test IO::Handle.nl-in can be set
• e82b798 Test IO::Handle.open respects attributes
• 2c29150 Test IO::Handle.nl-in attribute
• 03ce93b Test IO::Handle.encoding can be set
• 8ae81c0 Test no-arg candidate of &note
• fb61306 Test IO::Path.parts attribute
• 7266522 Test return type of IO::Spec::Unix.path
• 6ac3b4a Test IO::Spec::Win32.path
• dbbea15 Test IO::Handle.perl.EVAL roundtrips
• 5eb513c Test IO::Path.resolve sets CWD to $!SPEC.dir-sep
• b0c4a7a Test &words, IO::Handle.words, and IO::Path.words
• f3d1f67 Test $limit arg with &lines/IO::*.lines
• 4f5589b Add test for handle leak in IO::Path.lines
• 4d0f97a Add &put/IO::Handle.put tests
• 125fe18 Add &prompt tests
• 939ca8d Test IO::CatHandle.close
• 9833012 Test IO::CatHandle.get
• 2f65a72 Test IO::CatHandle.getc
• a4a7eaa Test IO::CatHandle.words
• 1131c09 Add &put/IO::Handle.put tests
• 80de9b6 Add &prompt tests
• bacfd9f Test IO::CatHandle.slurp
• e78e3c0 Test IO::CatHandle.comb/.split
• f1c1125 Test IO::CatHandle.read
• e9e78e1 Test IO::CatHandle.readchars
• 0479087 Test IO::CatHandle.Supply
• 71953e3 Test IO::CatHandle.encoding
• db4847e Test IO::CatHandle.eof
• 175ba45 Test IO::CatHandle.t/.path/.IO/.native-descriptor
• c6cc66a Test IO::CatHandle.gist/.Str/.opened/.open
• dcdac1a Test IO::CatHandle.lock/.unlock/.seek/.tell
• f48c26e Test IO::CatHandle.chomp/.nl-in
• 8afd758 Test IO::CatHandle.DESTROY
• c7eff2b Test IO::CatHandle.on-switch
• e87e20d Test IO::CatHandle.next-handle
• 28717f0 Test IO::CatHandle.perl method
• 432bf94 Test IO::Path.watch
• ce1b637 Test IO::Handle.say
• 0bb6298 Test IO::Handle.print-nl
• 47c88ab Test IO::Pipe.proc attribute
• 945621d Test IO::Path.SPEC attribute
• 5fb4b63 Test IO::Path.CWD/.path attributes
• d0e5701 Test IO::Path.Numeric and other .numeric methods
• 94d7133 Test 0-arg &say/&put/&print
• 38c61cd Test &slurp() and &slurp(IO::Handle)

Perl 6 Documentation Commits

I've made 146 commits to the Perl 6 Documentation repository:

• fd7a41b Improve code example
• 110efb4 No need for.ends-with``
• 69d32da Remove IO::Handle.z
• d02ae7d Remove IO::Handle.rw and .rwx
• ccae74a Fix incorrect information for IO::Path.absolute
• 3cf943d Expand IO::Path.relative
• cc496eb Remove mention of IO.umask
• 335a98d Remove mention ofrole IO``
• cc6539b Remove 8 methods from IO::Handle
• 0511e07 Document IO::Spec::*.tmpdir
• db36655 Remove tip to use $*SPEC to detect OS
• 839a6b3 Expand docs for $*HOME and $*TMPDIR
• d050d4b Remove IO::Path.chdir prose
• 1d0e433 Document &chdir
• 3fdc6dc Document &*chdir
• e1a299c Reword "defined as" for &*chdir
• e5225be Fix URL to &*chdir
• bf377c7 Document &indir
• 5aa614f Improve suggestion for Perl 5's opendir
• a53015a Clarify value of IO::Path.path
• bdd18f1 Fix desc of IO::Path.Str
• b78d4fd Include type names in links to methods
• b8fba97 Point out my $*CWD = chdir … is an error
• d5abceb Write docs for all spurt routines
• b9e692e Document new IO::Path.extension
• 65cc372 Document IO::Path.concat-with
• 24a6ea9 Toss all of the TODO methods in IO::Spec*
• 1f75ddc Document IO::Spec*.abs2rel
• cc62dd2 Kill IO::Path.abspath
• 1973010 Document IO::Path.ACCEPTS
• b3a9324 Expand/fix up IO::Path.accessed
• 1cd7de0 Fix up type graph
• 56256d0 Minor formatting improvements in IO::Special
• 184342c Document IO::Special.what
• 6bd0f98 Dissuade readers from using IO::Spec*
• 7afd9c4 Remove unrelated related classes
• a43ecb9 Document IO::Path's $.SPEC and $.CWD
• e9b6809 Document IO::Path::* subclasses
• 9102b51 Fix up IO::Path.basename
• 5c1d3b6 Document IO::Spec::Unix.basename
• a1cb80b Document IO::Spec::Win32.basename
• 28b6283 Document IO::Spec::*.canonpath
• 50e5565 Document IO::Spec::*.catdir and .catfile
• dbdc995 Document IO::Spec::*.catpath
• 0ca2295 Reword/expand IO::Path intro prose
• 45e84ad Move IO::Path.path to attributes
• b9de84f Remove DateTime tutorial from IO::Path docs
• 69b2082 Document IO::Path.chdir
• d436f3c Document IO::Spec::* don't do any validation
• 4090446 Improve chmod docs
• 1527d32 Document :completely arg to IO::Path.resolve
• 372545c Straighten up file test docs
• a30fae6 Avoid potential confusion with use of word "object"
• 2aa3c9f Document new behaviour of IO::Handle.Supply
• 56b50fe Document IO::Handle.slurp
• 017acd4 Improve docs for IO::Path.slurp
• 0f49bb5 List Rakudo-supported encodings in open()
• e60da5c List utf-* alias examples too since they're common
• f83f78c Use idiomatic Perl 6 in example
• fff866f Fix docs for symlink/link routines
• aeeec94 Straighten up copy, move, rename
• 923ea05 Straighten up mkdir docs
• 47b0526 Explicitly spell out caveats of IO::Path.Str
• 60b9227 Change return value formkdir``
• 8d95371 Expand IO::Handle/IO::Pipe.path docs
• fd8a5ed Document IO::Pipe.path
• bd4fa68 Document IO::Handle/IO::Pipe.IO
• 2aaf12a Document IO::Handle.Str
• 53f2b99 Documentrole IO's new purpose
• 160c6a2 Document IO::Handle.lock/.unlock
• 3145979 Document IO::Path.child-secure
• c5524ef Rename IO::Path.concat-with to .add
• 81a5806 Amend IO::Path.resolve: :completely
• 6ca67e4 Start sketching out Definitive IO Guide™
• b9c9117 Toss IO::Path.child-secure
• 61cb776 Document IO::Path.sibling
• 0fc39a6 Fix typegraph
• 9a63dc4 Document IO::Path.cleanup
• 2387ce3 Re-write IO::Handle.close docs
• 0def0d1 Amend IO::Handle.close docs
• c7e32e2 Document IO::Spec::Unix.curupdir
• fe489dc Document IO::Spec::Unix.curdir
• 83d5de0 Document IO::Spec::Unix.updir
• 4804128 Document IO::Handle.DESTROY
• c991862 Add warning to dir about...
• eca21ff Document copy/move behaviour for same target/source
• 6c2b8b2 Document IO::Path/IO::Handle.comb
• fb29e04 Include exception used in IO::Path.resolve
• 69d473f Document IO::Spec::*.devnull
• 994d671 List IO::Dir as one of the means...
• 4432ef3 Finish up IO::Path.dir docs
• 64355c8 Document IO::Spec::*.dir-sep
• 914c100 Finish up IO::Path.dirname
• 8d5e31c Document IO::Handle.encoding
• d5c36aa Finish off IO::Handle.eof
• e9de97e Document IO::Spec::*.extension
• bf7ec00 Document IO::Handle.flush
• 25bce38 Document IO::Path.succ
• 8233960 Improve IO::Handle.t docs
• b4006a2 Be explicit what IO::Handle.opened returns
• c4f27a7 Document IO::Path.pred
• 860333f Remove entirely-invented "File test operators"
• ab0bd7a Document IO::Path.Numeric/.Int
• 4f81f08 Improve IO::Handle.get docs
• c45d389 Finish off IO::Handle.getc/&getc docs
• a4012e0 Document IO::Handle.gist
• d15b0c7 Document IO::Path.gist
• 1cf6932 Document IO::Spec::*.is-absolute
• 4e88b84 Finish up IO::Path.is-absolute
• 497e7f7 Finish off IO::Path.is-relative
• f7e75c1 Document IO::Handle.nl-in
• e309ddd Finish up &note
• 81900cb Finish off IO::Path.parent
• 59cbc38 Finish off IO::Path.parts
• b99a666 Finish off IO::Path.path/.IO
• b070999 Document IO::Spec::*.path
• bace8ff Document IO::Path*.perl
• dfdd845 Add "The Basics" section to TDIOG
• cdc701e Add "What's an IO::Path Anyway?" section to TDIOG
• 0d6d058 Add "Writing into files" Section to TDIOG
• a6365f3 Document IO::Handle.words/&words
• 2e25c82 Document IO::Spec::*.join
• 49e58bd Document IO::Handle.lines
• 1744820 Document IO::Path.lines
• f3f70a0 Document IO::Path.words
• 509f0e8 Fix incorrect suggested routine
• a6f1cbf Fix up IO::Handle.print
• 8f53830 Fix up IO::Handle.print-nl
• dc50211 Fix &prompt
• 98965b3 Fix up IO::Handle.split
• bd702e2 Fix up IO::Handle.comb
• 6dd92b8 Document IO::CatHandle
• edeb069 Document IO::Path.split
• 2d96596 Document IO::Spec::*.split
• 129c097 Document IO::Spec::*.splitdir
• b946960 Document IO::Spec::*.splitpath
• dcd7490 Fix rmdir docs
• 2a7bd17 Document IO::Spec::*.rel2abs
• f45241f Document IO::Spec::*.rootdir
• 70a80ec Document IO::Handle.put
• 6f58ed0 Polish IO::Handle.say
• 3790a0f Polish &put/&print/&say
• ebb6f53 Document IO::Handle.nl-out attribute
• 53c9c91 Document IO::Handle.chomp attribute
• ca2a3a0 Improve &open/IO::Handle.open docs
• 856e846 Add Reading From Files section to TDIOG

This document is the April, 2017 progress report for TPF Standardization, Test Coverage, and Documentation of Perl 6 I/O Routines grant

Timing

As proposed to and approved by the Grant Manager, I've extended the due date for this grant by 1 extra month, in exchange for doing some extra optimization work on IO routines at no extra cost. The new completion date is May 22nd; right after the next Rakudo compiler release.

Communications

I've created and published three notices as part of this grant, informing the users on what is changing and how to best upgrade their code, where needed:

• Upgrade Information for Changes Due to IO Grant Work
• PART 2: Upgrade Information for Changes Due to IO Grant Work
• PART 3: Information on Changes Due to IO Grant Work

IO Action Plan Progress

Most of the IO Action Plan has been implemented and got shipped in Rakudo's 2017.04.2 release. The remaining items are:

• Implement better way to signal closed handle status (was omited from release due to original suggestion to do this not being ideal; likely better to do this on the VM end)
• Implement IO::CatHandle as a generalized IO::ArgFiles (was omited from release because it was decided to make it mostly-usable wherever IO::Handle can be used, and IO::ArgFiles is far from that, having only a handful of methods implemented)
• Optimization of the way we perform stat calls for multiple file tests (entirely internal that requires no changes to users' code)

Documentation and Coverage Progress

In my spreadsheet with all the IO routines and their candidates, the totals show that 40% have been documented and tested. Some of the remaining 60% may already have tests/docs added when implementing IO Action Plan or ealier and merely need checking and verification.

Optimizations

Some of the optimizations I promised to deliver in exchange for grant deadline extension were already done on IO::Spec::Unix and IO::Path routines and have made it into the 2017.04.2 release. Most of the optimizations that will be done in the upcoming month will be done in IO::Spec::Win32 and will largely affect Windows users.

IO Optimizations in 2017.04.2 Done by Other Core Members:

• Elizabeth Mattijsen made .slurp 2x faster rakudo/b4d80c0
• Samantha McVey made nqp::index—which is used in path operations—2x faster rakudo/f1fc879
• IO::Pipe.lines was made 3.2x faster by relying on work done by Elizabeth Mattijsen rakudo/0c62815

Tickets Resolved

The following tickets have been resolved as part of the grant:

• RT#130460 Can we relax indir's test on the target directory?: Resolved by making indir's default test to be only :d
• RT#130900: nul in pathname: Resolved by checking the path for nul and throwing when found
• RT#127407: [RFC] (1) add method IO::Path.stemname; (2) expand method IO::Path.parts: RFC rejected, but the described functionality is now available via the .extension method that was made much more powerful as part of IO Action Plan

Possibly more tickets were addressed by the IO Action Plan implementation, but they still need further review.

Bugs Fixed

• Fixed a bug in IO::Path.resolve with combiners tucked on the path separator. Fix in rakudo/9d8e391f3b; tests in roast/92217f75ce. The bug was identified by Timo Paulssen while testing secure implementation of IO::Path.child

IO Bug Fixes in 2017.04.2 Done by Other Core Members:

• Timo Paulssen fixed a bug with IO::Path types not being accepted by is native NativeCall trait rakudo/99840804
• Elizabeth Mattijsen fixed an issue in assignment to dynamics. This made it possible to temp $*TMPDIR variable rakudo/1b9d53
• Jonathan Worthington fixed a crash when slurping large files in binary mode with &slurp or IO::Path.slurp rakudo/d0924f1a2
• Jonathan Worthington fixed a bug with binary slurp reading zero bytes when another thread is causing a lot of GC rakudo/756877e

Commits

So far, I've commited 192 IO grant commits to rakudo/roast/doc repos.

Rakudo

69 IO grant commits:

• c6fd736 Make IO::Spec::Win32.is-absolute about 63x faster
• 7112a08 Add :D on invocant for file tests
• 8bacad8 Implement IO::Path.sibling
• a98b285 Remove IO::Path.child-secure
• 0b5a41b Rename IO::Path.concat-with to .add
• 9d8e391 Fix IO::Path.resolve with combiners; timotimo++
• 1887114 Implement IO::Path.child-secure
• b8458d3 Reword method child for cleaner code
• 51e4629 Amend rules for last part in IO::Path.resolve
• 9a2446c Move Bool return value to signature
• 214198b Implement proper args for IO::Handle.lock
• c95c4a7 Make IO::Path/IO::Special do IO role
• fd503f8 grant] Remove role IO and its .umask method"
• 0e36bb2 Make IO::Spec::Win32!canon-cat 2.3x faster
• 40217ed Swap .child to .concat-with in all the guts
• 490ffd1 Do not use self.Str in IO::Path errors
• 1f689a9 Fix up IO::Handle.Str
• c01ebea Make IO::Path.mkdir return invocant on success
• d46e8df Add IO::Pipe .path and .IO methods
• 6ee71c2 Coerce mode in IO::Path.mkdir to Int
• 0d9ecae Remove multi-dir &mkdir
• ff97083 Straighten up rename, move, and copy
• 7f73f92 Make IO::Path.new-from-absolute-path private
• da1dea2 Fix &symlink and &link
• 8c09c84 Fix symlink and link routines
• 90da80f Rework read methods in IO::Path/IO::Handle
• c13480c IO::Path.slurp: make 12%-35% faster; propagate Failures
• f1b4af7 Implement IO::Handle.slurp
• 184d499 Make IO::Handle.Supply respect handle's mode
• b6838ee Remove .f check in .z
• 6a8d63d Implement :completely param in IO::Path.resolve
• e681498 Make IO::Path throw when path contains NUL byte
• b4358af Delete code for IO::Spec::Win32.catfile
• 0a442ce Remove type constraint in IO::Spec::Cygwin.canonpath
• 0c8bef5 Implement :parent in IO::Spec::Cygwin.canonpath
• a0b82ed Make IO::Path::* actually instantiate a subclass
• 67f06b2 Run S32-io/io-special.t test file
• 954e69e Fix return value of IO::Special methods
• a432b3d Remove IO::Path.abspath (part 2)
• 94a6909 Clean up IO::Spec::Unix.abs2rel a bit
• 966a7e3 Implement IO::Path.concat-with
• 50aea2b Restore IO::Handle.IO
• 15a25da Fix ambiguity in empty extension vs no extension
• b1e7a01 Implement IO::Path.extension 2.0
• b62d1a7 Give $*TMPDIR a container
• 099512b Clean up and improve all spurt routines
• cb27bce Clean up &open and IO::Path.open
• 4c31903 Add S32-io/chdir-process.t to list of test files to run
• 5464b82 Improve &*chdir
• 2483d68 Fix regression in &chdir's failure mode
• ca1acb7 Fix race in &indir(IO::Path …)
• a0ef2ed Improve &chdir, &indir, and IO::Path.chdir
• aa62cd5 Remove &tmpdir and &homedir
• a5800a1 Implement IO::Handle.spurt
• 36ad92a Remove 15 methods from IO::Handle
• 87987c2 Remove role IO and its .umask method
• 9d8d7b2 Log all changes to plan made during review period
• 0c7e4a0 Do not capture args in .IO method
• c360ac2 Fix smartmatch of Cool ~~ IO::Path
• fa9aa47 Make R::I::SET_LINE_ENDING_ON_HANDLE 4.1x Faster
• 0111f10 Make IO::Spec::Unix.catdir 3.9x Faster
• 4fdebc9 Make IO::Spec::Unix.split 36x Faster
• dcf1bb2 Make IO::Spec::Unix.rel2abs 35% faster
• 55abc6d Improve IO::Path.child perf on *nix
• 4032953 Make IO::Handle.open 75% faster
• a01d679 Remove IO::Path.pipe
• 212cc8a Remove IO::Path.Bridge
• 76f7187 Do not cache IO::Path.e results
• dd4dfb1 Fix crash in IO::Special .WHICH/.Str

Perl 6 Specification

47 IO grant commits:

• 3b36d4d Test IO::Path.sibling
• 7a063b5 Fudge .child-secure tests
• 39677c4 IO::Path.concat-with got renamed to .add
• 92217f7 Test IO::Path.child-secure with combiners
• f3c5dae Test IO::Path.child-secure
• a716962 Amend rules for last part in IO::Path.resolve
• 4194755 Test IO::Handle.lock/.unlock
• 64ff572 Cover IO::Path/IO::Pipe's .Str/.path/.IO
• 637500d Spec IO::Pipe.path/.IO returns IO::Path type object
• 8fa49e1 Test link routines
• 416b746 Test symlink routines
• d4353b6 Rewrite .l on broken symlinks test
• 7e4a2ae Swap .slurp-rest to .slurp
• a4c53b0 Use bin IO::Handle to test its .Supply
• feecaf0 Expand file tests
• a809f0f Expand IO::Path.resolve tests
• ee7f05b Move is-path sub to top so it can be reused
• b16fbd3 Add tests to check nul byte is rejected
• 8f73ad8 Change \0 roundtrip test to \t roundtrip test
• 7c7fbb4 Cover :parent arg in IO::Spec::Cygwin.canonpath
• 896033a Cover IO::Spec::QNX.canonpath
• c3c51ed Cover IO::Spec::Win32.basename
• d8707e7 Cover IO::Spec::Unix.basename
• bd8d167 Test IO::Path::* instantiate a subclass
• 43ec543 Cover methods of IO::Special
• e5dc376 Expand IO::Path.accessed tests
• 0e47f25 Test IO::Path.concat-with
• 305f206 Test empty-string extensions in IO::Path.extension
• 2f09f18 Fix incorrect test
• b23e53e Test IO::Path.extension
• 1d4e881 Test $*TMPDIR can be temped
• 79ff022 Expand &spurt and IO::Path.spurt tests
• ba3e7be Merge S32-io/path.t and S32-io/io-path.t
• 3c4e81b Test IO::Path.Str works as advertised
• 86c5f9c Delete qp{} tests
• 430ab89 Test &*chdir
• 86f79ce Expand &chdir tests
• 73a5448 Remove two fudged &chdir tests
• 04333b3 Test &indir fails with non-existent paths by default
• bd46836 Amend &indir race tests
• f48198f Test &indir
• 5a7a365 Expand IO::Spec::*.tmpdir tests
• 14b6844 Use Numeric instead of IO role in dispatch test
• 8d6ca7a Cover IO::Path.ACCEPTS
• 091931a Expand &open tests
• 465795c Test IO::Path.lines(*) does not crash
• 63370fe Test IO::Special .WHICH/.Str do not crash

Documentation

76 IO grant commits:

• 61cb776 Document IO::Path.sibling
• b9c9117 Toss IO::Path.child-secure
• 6ca67e4 Start sketching out Definitive IO Guide™
• 81a5806 Amend IO::Path.resolve: :completely
• c5524ef Rename IO::Path.concat-with to .add
• 3145979 Document IO::Path.child-secure
• 160c6a2 Document IO::Handle.lock/.unlock
• 53f2b99 Document role IO's new purpose
• 2aaf12a Document IO::Handle.Str
• bd4fa68 Document IO::Handle/IO::Pipe.IO
• fd8a5ed Document IO::Pipe.path
• 8d95371 Expand IO::Handle/IO::Pipe.path docs
• 60b9227 Change return value for mkdir
• 47b0526 Explicitly spell out caveats of IO::Path.Str
• 923ea05 Straighten up mkdir docs
• aeeec94 Straighten up copy, move, rename
• fff866f Fix docs for symlink/link routines
• f83f78c Use idiomatic Perl 6 in example
• e60da5c List utf-* alias examples too since they're common
• 0f49bb5 List Rakudo-supported encodings in open()
• 017acd4 Improve docs for IO::Path.slurp
• 56b50fe Document IO::Handle.slurp
• 2aa3c9f Document new behaviour of IO::Handle.Supply
• a30fae6 Avoid potential confusion with use of word "object"
• 372545c Straighten up file test docs
• 1527d32 Document :completely arg to IO::Path.resolve
• 4090446 Improve chmod docs
• d436f3c Document IO::Spec::* don't do any validation
• 69b2082 Document IO::Path.chdir
• b9de84f Remove DateTime tutorial from IO::Path docs
• 45e84ad Move IO::Path.path to attributes
• 0ca2295 Reword/expand IO::Path intro prose
• dbdc995 Document IO::Spec::*.catpath
• 50e5565 Document IO::Spec::*.catdir and .catfile
• 28b6283 Document IO::Spec::*.canonpath
• a1cb80b Document IO::Spec::Win32.basename
• 5c1d3b6 Document IO::Spec::Unix.basename
• 9102b51 Fix up IO::Path.basename
• e9b6809 Document IO::Path::* subclasses
• a43ecb9 Document IO::Path's $.SPEC and $.CWD
• 7afd9c4 Remove unrelated related classes
• 6bd0f98 Dissuade readers from using IO::Spec*
• 184342c Document IO::Special.what
• 56256d0 Minor formatting improvements in IO::Special
• 1cd7de0 Fix up type graph
• b3a9324 Expand/fix up IO::Path.accessed
• 1973010 Document IO::Path.ACCEPTS
• cc62dd2 Kill IO::Path.abspath
• 1f75ddc Document IO::Spec*.abs2rel
• 24a6ea9 Toss all of the TODO methods in IO::Spec*
• 65cc372 Document IO::Path.concat-with
• b9e692e Document new IO::Path.extension
• d5abceb Write docs for all spurt routines
• b8fba97 Point out my $*CWD = chdir … is an error
• b78d4fd Include type names in links to methods
• bdd18f1 Fix desc of IO::Path.Str
• a53015a Clarify value of IO::Path.path
• 5aa614f Improve suggestion for Perl 5's opendir
• bf377c7 Document &indir
• e5225be Fix URL to &*chdir
• e1a299c Reword "defined as" for &*chdir
• 3fdc6dc Document &*chdir
• 1d0e433 Document &chdir
• d050d4b Remove IO::Path.chdir prose
• 839a6b3 Expand docs for $*HOME and $*TMPDIR
• db36655 Remove tip to use $*SPEC to detect OS
• 0511e07 Document IO::Spec::*.tmpdir
• cc6539b Remove 8 methods from IO::Handle
• 335a98d Remove mention of role IO
• cc496eb Remove mention of IO.umask
• 3cf943d Expand IO::Path.relative
• ccae74a Fix incorrect information for IO::Path.absolute
• d02ae7d Remove IO::Handle.rw and .rwx
• 69d32da Remove IO::Handle.z
• 110efb4 No need for .ends-with
• fd7a41b Improve code example

This document is the March, 2017 progress report for TPF Standardization, Test Coverage, and Documentation of Perl 6 I/O Routines grant

Timing

My delivery of the Action Plan was one week later than I originally expected to deliver it. The delay let me assess some of the big-picture consistency issues, which led to proposal to remove 15 methods from IO::Handle and to iron out naming and argument format for several other routines.

I still hope to complete all the code modifications prior to end of weekend of April 15, so all of these can be included in the next Rakudo Star release. And a week after, I plan to complete the grant.

Note: to minimize user impact, some of the changes may be included only in 6.d language, which will be available in 2017.04 release only if the user uses use v6.d.PREVIEW pragma.

IO Action Plan

I finished the IO Action Plan, placed it into /doc of rakudo's repository, and made it available to other core devs for review for a week (period ends on April 1st). The Action Plan is 16 pages long and contains 26 sections detailing proposed changes.

Overall, I proposed many more smaller changes than I originally expected and fewer larger, breaking changes than I originally expected. This has to do with a much better understanding of how rakudo's IO routines are "meant to" be used, so I think the improvement of the documentation under this grant will be much greater than I originally anticipated.

A lot of this has to do with lack of explanatory documentation for how to manipulate and traverse paths. This had the effect that users were using the $*SPEC object (157 instances of its use in the ecosystem!) and its routines for that goal, which is rather awkward. This concept is prevalent enough that I even wrote SPEC::Func module in the past, due to user demand, and certain books whose draft copies I read used the $*SPEC as well.

In reality, $*SPEC is an internal-ish thing and unless you're writing your own IO abstractions, you never need to use it directly. The changes and additions to the IO::Path methods done under this grant will make traversing paths even more pleasant, and the new tutorial documentation I plan to write under this grant will fully describe the Right Way™ to do it all.

In fact, removal of $*SPEC in future language versions is currently under consideration...

Removal of $*SPEC

lizmat++ pointed out that we can gain significant performance improvements by removing $*SPEC infrastructure and moving it into module-space. For example, a benchmark of slurping a 10-line file shows that removal of all the path processing code makes benched program run more than 14x faster. When benching IO::Path creation, dynamic var lookup alone takes up 14.73% of the execution time.

The initial plan was to try and make IO routines handle all OSes in a unified way (e.g. using / on Windows), however it was found this would create several ambiguities and would be buggy, even if fast.

However, I think there are still a lot of improvements that can be gained by making $*SPEC infrastructure internal. So we'd still have the IO::Spec-type modules but they'll have a private API we can optimize freely, and we'll get rid of the dynamic lookups, consolidate what code we can into IO::Path, while keeping the functionality that differs between OSes in the ::Spec modules.

Since this all sounds like guestimation and there's a significant-ish use of $*SPEC in the ecosystem, the plan now is to implement it all in a module first and see whether it works well and offers any significant performance improvements. If it does, I believe it should be possible to swap IO::Path to use the fast version in 6.d language, while still leaving $*SPEC dynvar and its modules in core, as deprecated, for removal in 6.e.

This won't be done under this grant, and while trying not to over-promise, I hope to release this module some time in May-June. So keep an eye out for it; I already picked out a name: FastIO

newio Branch

As per original goals of the grant, I reviewed the code in Rakudo's 2014–2015 newio branch, to look for any salvagable ideas. I did not have any masterplan design documents to go with it and I tried a few commits but did not find one that didn't have merge conflicts and compiled (it kept complaining about ModuleLoader), so my understanding of it comes solely from reading the source code, and may be off from what the original author intended it to be.

The major difference between newio and Rakudo's current IO structure is type hierarchy and removal of $*SPEC. newio provides IO::Pathy and PIO roles which are done by IO::File, IO::Dir, IO::Local, IO::Dup, IO::Pipe, and IO::Huh classes that represent various IO objects. The current Rakudo's system has fewer abstractions: IO::Path represents a path to an IO object and IO::Handle provides read/write access to it, with IO::Pipe handling pipes, and no special objects for directories (their contents are obtained via IO::Path.dir method and their attributes are modified via IO::Path methods).

Since 6.d language is additive to 6.c language, completely revamping the type hierarchy may be challenging and messy. I'm also not entirely sold on what appears to be one of the core design ideas in newio: most of the abstractions are of IO objects as they were at the object instantiation time. An IO::Pathy object represents an IO item that exists, despite there being no guarantees that it actually does. Thus, IO::File's .f and .e methods always return True, while its .d method always returns False. This undoubtedly gives a performance enhancement, however, if $ rm foo were executed after IO::File object's creation, the .e method would no longer return correct data and if then $ mkdir foo were executed, both .f and .d methods would be returning incorrect data.

Until recently, Rakudo cached the result of .e call and that produced unexpected by user behaviour. I think the issue will be greatly exacerbated if this sort of caching is extended to entire objects and many of their methods.

However, I do think the removal of $*SPEC is a good idea. And as described in previous section I will try to make a FastIO module, using ideas from newio branch, for possible inclusion in future language versions.

Experimental MoarVM Coverage Reporter

As was mentioned in my grant proposal, the coverage reporter was busted by the upgrade of information returned by .file and .line methods on core routines. MasterDuke++ made several commits fixing numerous issues to the coverage parser and last night I identified the final piece of the breakage. The annotations and hit reports all use the new SETTING::src/core/blah file format. The setting file has SETTING::src/core/blah markers inside of it. The parser however, still thinks it's being fed the old gen/moar/CORE.setting filenames, so once I teach it to calculate proper offsets into the setting file, we'll have coverage reports on perl6.wtf back up and running and I'll be able to use them to judge IO routine test coverage required for this grant.

Performance Improvements

Although not planned by the original grant, I was able to make the following performance enhancements to IO routines. So hey! Bonus deliverables \o/:

• rakudo/fa9aa47 Make R::I::SET_LINE_ENDING_ON_HANDLE 4.1x Faster
• rakudo/0111f10 Make IO::Spec::Unix.catdir 3.9x Faster
• rakudo/4fdebc9 Make IO::Spec::Unix.split 36x Faster
  • Affects IO::Path's .parent, .parts, .volume, .dirname, and .basename
  • Measurement of first call to .basename shows it's now 6x-10x faster
• rakudo/dcf1bb2 Make IO::Spec::Unix.rel2abs 35% faster
• rakudo/55abc6d Improve IO::Path.child perf on *nix:
  • make IO::Path.child 2.1x faster on *nix
  • make IO::Spec::Unix.join 8.5x faster
  • make IO::Spec::Unix.catpath 9x faster
• rakudo/4032953 Make IO::Handle.open 75% faster
• rakudo/4eef6db Make IO::Spec::Unix.is-absolute about 4.4x faster
• rakudo/ae5e510 Make IO::Path.new 7% faster when creating from Str
• rakudo/0c6281 Make IO::Pipe.lines use IO::Handle.lines for 3.2x faster performance

Performance Improvements Made By Other Core Developers

lizmat++ also made these improvements in IO area:

• rakudo/b4d80c0 Make .IO.slurp about 2x as fast
• rakudo/9da50e3 Introducing IO::Handle.iterator
• rakudo/9019a5b Streamline IO::Handle.get/getc
• rakudo/4bc826d Streamline IO::Handle.get

Along with the commits above, she also made IO::Handle.lines faster and eliminated a quirk that required custom .lines implementation in IO::Pipe (which is a subclass of IO::Handle). Due to that, I was able to remove old IO::Pipe.lines implementation and make it use new-and-improved IO::Handle.lines, which made the method about 3.2x faster.

Bugs

Will (attempt to) fix as part of the grant

• IO::Pipe inherits .t method from from IO::Handle to check if the handle is a TTY, however, attempt to call it causes a segfault. MasterDuke++ already found the candidate for the offending code (MoarVM/Issue#561) and this should be resolved by the time this grant is completed.

Don't think I will be able to fix these as part of the grant

• Found a strange error generated when IO::Pipe's buffer is filled up. This is too deep in the guts for me to know how to resolve yet, so I filed it as RT#131026

Already Fixed

• Found that IO::Path had a vestigial .pipe method that delegated to a non-existant IO::Handle method. Removed in rakudo/a01d67
• Fixed IO::Pipe.lines not accepting a Whatever as limit, which is accepted by all other .lines. rakudo/0c6281 Tests in roast/465795 and roast/add852
• Fixed issues due to caching of IO::Handle.e. Reported as RT#130889. Fixed in rakudo/76f718. Tests in roast/908348
• Rejected rakudo PR#666 and resolved RT#126262 by explaining why the methods return Str objects instead of IO::Path on ticket/PR and improving the documentation by fixing mistakes (doc/ccae74) and expanding (doc/3cf943) on what the methods do exactly.
• IO::Path.Bridge was defunct, as it was trying to call .Bridge on Str, which does not exist. Resolved the issue by deleting this method in rakudo/212cc8
• Per demand, made IO::Path.dir a multi, so module-space can augment it with other candidates that add more functionality. rakudo/fbe7ace

This document is the February, 2017 progress report for TPF Standardization, Test Coverage, and Documentation of Perl 6 I/O Routines grant

Timing

I'm currently running slightly behind the schedule outlined in the grant. I expect to complete the Action Plan and have it ratified by other core members by March 18th, which is the date of the 2017.03 compiler release. Then, I'll implement all of the Action Plan (and complete the grant) by the 2017.04 compiler release on April 15th. This is also the release the next Rakudo Star distribution will be based on, and so the regular end users will receive better IO there and then.

Some members of the Core Team voiced concerns over implementing any changes that can break users' code, even if the changes do not break 6.c-errata specification tests. Once the full set of changes is known, they will be reviewed on a case-by-case basis, and some of them may be implemented under 6.d.PREVIEW pragma, to be included in 6.d language version, leaving 6.c language versions untouched. Note that changes that are decided to be 6.d material may delay the completion of this grant due to not fully-fleshed out infrastructure for supporting multiple language versions. The April 15th deadline stated above applies only to changes to 6.c language and new deadline will be ascertained for completion of the 6.d changes.

User Communications

I wrote and disseminated advanced notice of the changes to be made due to this grant, to prepare the users to expect some code to break (some routines were found to be documented, despite being absent entirely from the Specification and not officially part of the language).

The notice can be seen at: http://rakudo.org/2017/02/26/advance-notice-of-significant-changes/

It is possible the Core Team will decide to defer all breaking changes to 6.d language version, to be currently implemented under v6.d.PREVIEW pragma.

Bonus Deliverable

The bonus deliverable—The Map of Perl 6 Routines—is now usable. The code is available in perl6/routine-map repository, and the rendered version is available on map.perl6.party. Its current state is sufficient to serve the intended purpose for this grant, but I'll certainly add improvements to it sometime in the future, such as linking to docs, linking to routines' source code, having an IRC bot looking stuff up in it, etc.

It'll also be fairy easy to use the Map to detect undocumented routines or ones that are documented under the incorrect type.

Identified Issues/Deficiencies with IO Routines

These points, issues, and ideas were identified this month and will be included for consideration in the Action Plan.

• Calling practically any method on a closed IO::Handle results in an LTA (Less Than Awesome) error message that reads <something> requires an object with REPR MVMOSHandle where <something> is sometimes the name of the method called by the user and others is some internal method invoked indirectly. We need better errors for closed file handles; and not something that would require a is-fh-closed() type of conditional called in all the methods, which would be a hefty performance hit.
• Several routines have been identified which in other languages return useful information: number of bytes actually written or current file position, whereas in Perl 6 they just return a Bool (.print, .say, .write) or a Mu type object (.seek). Inconsistently, .printf does appear to return the number of bytes written. It should be possible to make other routines similarly useful, although I suspect some of it may have to wait until 6.d language release.
• The .seek routine takes the seek location as one of three Enum values. Not only are they quite lengthy to type, they're globally available for no good reason and .seek is virtually unique in using this calling convention. I will seek to standardize this routine to take mutually-exclusive named arguments instead, preferably with much shorter names, but those are yet to be bikeshed.
• IO.umask routine simply shells out to umask. This fails terribly on OSes that don't have that command, especially since the code still tries to decode the received input as an octal string, even after the failure. Needs improvement.
• link's implementation and documentation confuses what a "target" is. Luckily (or sadly?) there are exactly zero tests for this routine in the Perl 6 Specification, so we can change it to match the behaviour of ln Linux command and the foo $existing-thing, $new-thing argument order of move, rename, and other similar routines.
• When using run(:out, 'some-non-existant-command').out.slurp-rest it will silently succeed and return an empty string. If possible, this should be changed to return the failure or throw at some point.
• chdir's :test parameter for directory permissions test is taken as a single string parameter. This makes it extremely easy to mistakenly write broken code: for example, "/".IO.chdir: "tmp", :test<r w> succeeds, while "/".IO.chdir: "tmp", :test<w r> fails with a misleading error message saying the directory is not readable/writable. I will propose for :test parameter to be deprecated in favour of using multiple named arguments to indicate desired tests. By extension, similar change will be applied to indir, tmpdir, and homedir routines (if they remain in the language).
• Documentation: several inaccuracies in the documentation were found. I won't be identifying these in my reports/Action Plan, but will simply ensure the documentation matches the implementation once the Action Plan is fully implemented.

Discovered Bugs

The hunt for 6-legged friends has these finds so far:

Will (attempt to) fix as part of the grant

• indir() has a race condition where the actual dir it runs in ends up being wrong. Using indir '/tmp/useless', { qx/rm -fr */ } in one thread and backing up your precious data in another has the potential to result in some spectacular failurage.
• perl6 -ne '@ = lines' crashes after first iteration, crying about MVMOSHandle REPR. I suspect the code is failing to follow iterator protocol somewhere and is attempting to read on an already closed handle. I expect to be able to resolve this and the related RT#128047 as part of the grant.
• .tell incorrectly always returns 0 on files opened in append mode
• link mixes up target and link name in its error message

Don't think I will be able to fix these as part of the grant

• seek() with SeekFromCurrent as location fails to seek correctly if called after .readchars, but only on MoarVM. This appears to occur due to some sort of buffering. I filed this as RT#130843.
• On JVM, .readchars incorrectly assumes all chars are 2 bytes long. This appears to be just a naive substitute for nqp::readcharsfh op. I filed this as RT#130840.

Already Fixed

• While making the Routine Map, I discovered .WHICH and .Str methods on IO::Special were only methods defined only for the :D subtype, resulting in a crash when using, say, infix:<eqv> operator on the type object, instead Mu.WHICH/.Str candidates getting invoked. This bug was easy and I already commited fix in radudo/dd4dfb14d3 and tests to cover it in roast/63370fe054

Auxiliary Bugs

While doing the work for this grant, I also discovered some non-IO related bugs (some of which I fixed):

• .Bool, .so, .not, .hash, and .elems on Baggy:U crash (fixed in e8af8553)
• .sort on reified empty arrays crashes (reported as RT#130866)
• SEGV with Scalar type object in unique() (reported as RT#130852)
• .Bool, .so, .not and possibly others crash on Seq:U; need to evaluate entire codebase to see where only methods are used instead of multi, preventing dispatch to Mu.* candidates. (reported as RT#130867 )
• Incorrect line number reported for wrong routine call when unpacking/heredocs are used (reported as RT#130862)
• Warnings produced by core code marked as "shouldn't happen" (reported as RT#130857)