This document is the March, 2017 progress report for TPF Standardization, Test Coverage, and Documentation of Perl 6 I/O Routines grant

Timing

My delivery of the Action Plan was one week later than I originally expected to deliver it. The delay let me assess some of the big-picture consistency issues, which led to proposal to remove 15 methods from IO::Handle and to iron out naming and argument format for several other routines.

I still hope to complete all the code modifications prior to end of weekend of April 15, so all of these can be included in the next Rakudo Star release. And a week after, I plan to complete the grant.

Note: to minimize user impact, some of the changes may be included only in 6.d language, which will be available in 2017.04 release only if the user uses use v6.d.PREVIEW pragma.

IO Action Plan

I finished the IO Action Plan, placed it into /doc of rakudo's repository, and made it available to other core devs for review for a week (period ends on April 1st). The Action Plan is 16 pages long and contains 26 sections detailing proposed changes.

Overall, I proposed many more smaller changes than I originally expected and fewer larger, breaking changes than I originally expected. This has to do with a much better understanding of how rakudo's IO routines are "meant to" be used, so I think the improvement of the documentation under this grant will be much greater than I originally anticipated.

A lot of this has to do with lack of explanatory documentation for how to manipulate and traverse paths. This had the effect that users were using the $*SPEC object (157 instances of its use in the ecosystem!) and its routines for that goal, which is rather awkward. This concept is prevalent enough that I even wrote SPEC::Func module in the past, due to user demand, and certain books whose draft copies I read used the $*SPEC as well.

In reality, $*SPEC is an internal-ish thing and unless you're writing your own IO abstractions, you never need to use it directly. The changes and additions to the IO::Path methods done under this grant will make traversing paths even more pleasant, and the new tutorial documentation I plan to write under this grant will fully describe the Right Way™ to do it all.

In fact, removal of $*SPEC in future language versions is currently under consideration...

Removal of $*SPEC

lizmat++ pointed out that we can gain significant performance improvements by removing $*SPEC infrastructure and moving it into module-space. For example, a benchmark of slurping a 10-line file shows that removal of all the path processing code makes benched program run more than 14x faster. When benching IO::Path creation, dynamic var lookup alone takes up 14.73% of the execution time.

The initial plan was to try and make IO routines handle all OSes in a unified way (e.g. using / on Windows), however it was found this would create several ambiguities and would be buggy, even if fast.

However, I think there are still a lot of improvements that can be gained by making $*SPEC infrastructure internal. So we'd still have the IO::Spec-type modules but they'll have a private API we can optimize freely, and we'll get rid of the dynamic lookups, consolidate what code we can into IO::Path, while keeping the functionality that differs between OSes in the ::Spec modules.

Since this all sounds like guestimation and there's a significant-ish use of $*SPEC in the ecosystem, the plan now is to implement it all in a module first and see whether it works well and offers any significant performance improvements. If it does, I believe it should be possible to swap IO::Path to use the fast version in 6.d language, while still leaving $*SPEC dynvar and its modules in core, as deprecated, for removal in 6.e.

This won't be done under this grant, and while trying not to over-promise, I hope to release this module some time in May-June. So keep an eye out for it; I already picked out a name: FastIO

newio Branch

As per original goals of the grant, I reviewed the code in Rakudo's 2014–2015 newio branch, to look for any salvagable ideas. I did not have any masterplan design documents to go with it and I tried a few commits but did not find one that didn't have merge conflicts and compiled (it kept complaining about ModuleLoader), so my understanding of it comes solely from reading the source code, and may be off from what the original author intended it to be.

The major difference between newio and Rakudo's current IO structure is type hierarchy and removal of $*SPEC. newio provides IO::Pathy and PIO roles which are done by IO::File, IO::Dir, IO::Local, IO::Dup, IO::Pipe, and IO::Huh classes that represent various IO objects. The current Rakudo's system has fewer abstractions: IO::Path represents a path to an IO object and IO::Handle provides read/write access to it, with IO::Pipe handling pipes, and no special objects for directories (their contents are obtained via IO::Path.dir method and their attributes are modified via IO::Path methods).

Since 6.d language is additive to 6.c language, completely revamping the type hierarchy may be challenging and messy. I'm also not entirely sold on what appears to be one of the core design ideas in newio: most of the abstractions are of IO objects as they were at the object instantiation time. An IO::Pathy object represents an IO item that exists, despite there being no guarantees that it actually does. Thus, IO::File's .f and .e methods always return True, while its .d method always returns False. This undoubtedly gives a performance enhancement, however, if $ rm foo were executed after IO::File object's creation, the .e method would no longer return correct data and if then $ mkdir foo were executed, both .f and .d methods would be returning incorrect data.

Until recently, Rakudo cached the result of .e call and that produced unexpected by user behaviour. I think the issue will be greatly exacerbated if this sort of caching is extended to entire objects and many of their methods.

However, I do think the removal of $*SPEC is a good idea. And as described in previous section I will try to make a FastIO module, using ideas from newio branch, for possible inclusion in future language versions.

Experimental MoarVM Coverage Reporter

As was mentioned in my grant proposal, the coverage reporter was busted by the upgrade of information returned by .file and .line methods on core routines. MasterDuke++ made several commits fixing numerous issues to the coverage parser and last night I identified the final piece of the breakage. The annotations and hit reports all use the new SETTING::src/core/blah file format. The setting file has SETTING::src/core/blah markers inside of it. The parser however, still thinks it's being fed the old gen/moar/CORE.setting filenames, so once I teach it to calculate proper offsets into the setting file, we'll have coverage reports on perl6.wtf back up and running and I'll be able to use them to judge IO routine test coverage required for this grant.

Performance Improvements

Although not planned by the original grant, I was able to make the following performance enhancements to IO routines. So hey! Bonus deliverables \o/:

• rakudo/fa9aa47 Make R::I::SET_LINE_ENDING_ON_HANDLE 4.1x Faster
• rakudo/0111f10 Make IO::Spec::Unix.catdir 3.9x Faster
• rakudo/4fdebc9 Make IO::Spec::Unix.split 36x Faster
  • Affects IO::Path's .parent, .parts, .volume, .dirname, and .basename
  • Measurement of first call to .basename shows it's now 6x-10x faster
• rakudo/dcf1bb2 Make IO::Spec::Unix.rel2abs 35% faster
• rakudo/55abc6d Improve IO::Path.child perf on *nix:
  • make IO::Path.child 2.1x faster on *nix
  • make IO::Spec::Unix.join 8.5x faster
  • make IO::Spec::Unix.catpath 9x faster
• rakudo/4032953 Make IO::Handle.open 75% faster
• rakudo/4eef6db Make IO::Spec::Unix.is-absolute about 4.4x faster
• rakudo/ae5e510 Make IO::Path.new 7% faster when creating from Str
• rakudo/0c6281 Make IO::Pipe.lines use IO::Handle.lines for 3.2x faster performance

Performance Improvements Made By Other Core Developers

lizmat++ also made these improvements in IO area:

• rakudo/b4d80c0 Make .IO.slurp about 2x as fast
• rakudo/9da50e3 Introducing IO::Handle.iterator
• rakudo/9019a5b Streamline IO::Handle.get/getc
• rakudo/4bc826d Streamline IO::Handle.get

Along with the commits above, she also made IO::Handle.lines faster and eliminated a quirk that required custom .lines implementation in IO::Pipe (which is a subclass of IO::Handle). Due to that, I was able to remove old IO::Pipe.lines implementation and make it use new-and-improved IO::Handle.lines, which made the method about 3.2x faster.

Bugs

Will (attempt to) fix as part of the grant

• IO::Pipe inherits .t method from from IO::Handle to check if the handle is a TTY, however, attempt to call it causes a segfault. MasterDuke++ already found the candidate for the offending code (MoarVM/Issue#561) and this should be resolved by the time this grant is completed.

Don't think I will be able to fix these as part of the grant

• Found a strange error generated when IO::Pipe's buffer is filled up. This is too deep in the guts for me to know how to resolve yet, so I filed it as RT#131026

Already Fixed

• Found that IO::Path had a vestigial .pipe method that delegated to a non-existant IO::Handle method. Removed in rakudo/a01d67
• Fixed IO::Pipe.lines not accepting a Whatever as limit, which is accepted by all other .lines. rakudo/0c6281 Tests in roast/465795 and roast/add852
• Fixed issues due to caching of IO::Handle.e. Reported as RT#130889. Fixed in rakudo/76f718. Tests in roast/908348
• Rejected rakudo PR#666 and resolved RT#126262 by explaining why the methods return Str objects instead of IO::Path on ticket/PR and improving the documentation by fixing mistakes (doc/ccae74) and expanding (doc/3cf943) on what the methods do exactly.
• IO::Path.Bridge was defunct, as it was trying to call .Bridge on Str, which does not exist. Resolved the issue by deleting this method in rakudo/212cc8
• Per demand, made IO::Path.dir a multi, so module-space can augment it with other candidates that add more functionality. rakudo/fbe7ace

This document is the February, 2017 progress report for TPF Standardization, Test Coverage, and Documentation of Perl 6 I/O Routines grant

Timing

I'm currently running slightly behind the schedule outlined in the grant. I expect to complete the Action Plan and have it ratified by other core members by March 18th, which is the date of the 2017.03 compiler release. Then, I'll implement all of the Action Plan (and complete the grant) by the 2017.04 compiler release on April 15th. This is also the release the next Rakudo Star distribution will be based on, and so the regular end users will receive better IO there and then.

Some members of the Core Team voiced concerns over implementing any changes that can break users' code, even if the changes do not break 6.c-errata specification tests. Once the full set of changes is known, they will be reviewed on a case-by-case basis, and some of them may be implemented under 6.d.PREVIEW pragma, to be included in 6.d language version, leaving 6.c language versions untouched. Note that changes that are decided to be 6.d material may delay the completion of this grant due to not fully-fleshed out infrastructure for supporting multiple language versions. The April 15th deadline stated above applies only to changes to 6.c language and new deadline will be ascertained for completion of the 6.d changes.

User Communications

I wrote and disseminated advanced notice of the changes to be made due to this grant, to prepare the users to expect some code to break (some routines were found to be documented, despite being absent entirely from the Specification and not officially part of the language).

The notice can be seen at: http://rakudo.org/2017/02/26/advance-notice-of-significant-changes/

It is possible the Core Team will decide to defer all breaking changes to 6.d language version, to be currently implemented under v6.d.PREVIEW pragma.

Bonus Deliverable

The bonus deliverable—The Map of Perl 6 Routines—is now usable. The code is available in perl6/routine-map repository, and the rendered version is available on map.perl6.party. Its current state is sufficient to serve the intended purpose for this grant, but I'll certainly add improvements to it sometime in the future, such as linking to docs, linking to routines' source code, having an IRC bot looking stuff up in it, etc.

It'll also be fairy easy to use the Map to detect undocumented routines or ones that are documented under the incorrect type.

Identified Issues/Deficiencies with IO Routines

These points, issues, and ideas were identified this month and will be included for consideration in the Action Plan.

• Calling practically any method on a closed IO::Handle results in an LTA (Less Than Awesome) error message that reads <something> requires an object with REPR MVMOSHandle where <something> is sometimes the name of the method called by the user and others is some internal method invoked indirectly. We need better errors for closed file handles; and not something that would require a is-fh-closed() type of conditional called in all the methods, which would be a hefty performance hit.
• Several routines have been identified which in other languages return useful information: number of bytes actually written or current file position, whereas in Perl 6 they just return a Bool (.print, .say, .write) or a Mu type object (.seek). Inconsistently, .printf does appear to return the number of bytes written. It should be possible to make other routines similarly useful, although I suspect some of it may have to wait until 6.d language release.
• The .seek routine takes the seek location as one of three Enum values. Not only are they quite lengthy to type, they're globally available for no good reason and .seek is virtually unique in using this calling convention. I will seek to standardize this routine to take mutually-exclusive named arguments instead, preferably with much shorter names, but those are yet to be bikeshed.
• IO.umask routine simply shells out to umask. This fails terribly on OSes that don't have that command, especially since the code still tries to decode the received input as an octal string, even after the failure. Needs improvement.
• link's implementation and documentation confuses what a "target" is. Luckily (or sadly?) there are exactly zero tests for this routine in the Perl 6 Specification, so we can change it to match the behaviour of ln Linux command and the foo $existing-thing, $new-thing argument order of move, rename, and other similar routines.
• When using run(:out, 'some-non-existant-command').out.slurp-rest it will silently succeed and return an empty string. If possible, this should be changed to return the failure or throw at some point.
• chdir's :test parameter for directory permissions test is taken as a single string parameter. This makes it extremely easy to mistakenly write broken code: for example, "/".IO.chdir: "tmp", :test<r w> succeeds, while "/".IO.chdir: "tmp", :test<w r> fails with a misleading error message saying the directory is not readable/writable. I will propose for :test parameter to be deprecated in favour of using multiple named arguments to indicate desired tests. By extension, similar change will be applied to indir, tmpdir, and homedir routines (if they remain in the language).
• Documentation: several inaccuracies in the documentation were found. I won't be identifying these in my reports/Action Plan, but will simply ensure the documentation matches the implementation once the Action Plan is fully implemented.

Discovered Bugs

The hunt for 6-legged friends has these finds so far:

Will (attempt to) fix as part of the grant

• indir() has a race condition where the actual dir it runs in ends up being wrong. Using indir '/tmp/useless', { qx/rm -fr */ } in one thread and backing up your precious data in another has the potential to result in some spectacular failurage.
• perl6 -ne '@ = lines' crashes after first iteration, crying about MVMOSHandle REPR. I suspect the code is failing to follow iterator protocol somewhere and is attempting to read on an already closed handle. I expect to be able to resolve this and the related RT#128047 as part of the grant.
• .tell incorrectly always returns 0 on files opened in append mode
• link mixes up target and link name in its error message

Don't think I will be able to fix these as part of the grant

• seek() with SeekFromCurrent as location fails to seek correctly if called after .readchars, but only on MoarVM. This appears to occur due to some sort of buffering. I filed this as RT#130843.
• On JVM, .readchars incorrectly assumes all chars are 2 bytes long. This appears to be just a naive substitute for nqp::readcharsfh op. I filed this as RT#130840.

Already Fixed

• While making the Routine Map, I discovered .WHICH and .Str methods on IO::Special were only methods defined only for the :D subtype, resulting in a crash when using, say, infix:<eqv> operator on the type object, instead Mu.WHICH/.Str candidates getting invoked. This bug was easy and I already commited fix in radudo/dd4dfb14d3 and tests to cover it in roast/63370fe054

Auxiliary Bugs

While doing the work for this grant, I also discovered some non-IO related bugs (some of which I fixed):

• .Bool, .so, .not, .hash, and .elems on Baggy:U crash (fixed in e8af8553)
• .sort on reified empty arrays crashes (reported as RT#130866)
• SEGV with Scalar type object in unique() (reported as RT#130852)
• .Bool, .so, .not and possibly others crash on Seq:U; need to evaluate entire codebase to see where only methods are used instead of multi, preventing dispatch to Mu.* candidates. (reported as RT#130867 )
• Incorrect line number reported for wrong routine call when unpacking/heredocs are used (reported as RT#130862)
• Warnings produced by core code marked as "shouldn't happen" (reported as RT#130857)

Deconfusion note: if you're just a regular Perl 6 user, you likely use and only ever heard of Rakudo Star, which is a distribution that includes the Rakudo Perl 6 Compiler, some modules, and the docs. This post details a release of that compiler only, which gets released more often than Rakudo Star. So please don't think there's a new release, if Star is all you use.

Part I: Humans Make Errors

Today is the third Saturday of the month, which is an awesome day! It's when the Rakudo Perl 6 Compiler sees its monthly release. I was at the helm today, so I chugged along through the Rakudo release guide and the NQP release guide, whose release is part of the process as well.

We're All Green To Go

As I was nearing the end of the process, the first hint of a problem surfaced when a user joined the #perl6-dev IRC channel:

<nwc10> the tag 2016.08 seems to be missing from the nqp github repository
<Zoffix> nwc10, it was created about 44 minutes ago. Ensure you got the
latest everything
<nwc10> very strange. because if I pull again in my nqp checkout I still don't see it
<Zoffix> Did you add --tags? it's git pull --tags or git fetch --tags or something like that
<nwc10> Zoffix: correct. no I didn't. thanks
<Zoffix> \o/
<nwc10> OK, why do I usually not need to do that?

Good question. I pulled in a fresh checkout and ran the full test suite again, just to be sure. Everything passed. Luckily, I got the answer rather quickly:

<nwc10> git log --graph --decorate origin/master 2016.08
<nwc10> how that it and master have diverged
<nwc10> the tag is for a commit which is not a parent of nqp master

Here's the story: I have a local NQP checkout. I bump its version and its MoarVM dependency. Tag the release. And start the full build and test suite run. That's a lengthly process, so while that is happening, I go to GitHub and use the online editor to make a minor tweak to the text of the NQP's release guide.

The tests finish. I try to push my tag and the version bump and get the usual error saying the online repo has diverged. Instinctively, I run my gr alias for git pull --rebase to bring in the new changes, and then push all of the work into the repo.

The problem is that --rebase doesn't move my tag, so it's still referencing that old commit. If you clone the repo, everything appears to work for that tag, but if you pull in the changes into the existing repo, you don't get the tag. Also, git describe (which we use when doing mid-release NQP/MoarVM version bumps) was now missing my tag, because the commit my tag tagged is not the parent of the master HEAD.

At this point, I don't yet know about the breakage for folks who are doing git pull in an existing branch, and so I continue with the release, since all of my tests are green. Finish up. Then relax by departing to fly in No Man's Sky. Awesome!

The Robocide

The first hint (wait, the second hint?) of trouble showed up when the latest commits did not appear in our eval bot that's supposed to update itself to HEAD:

<Zoffix> m: dd [ $*VM.version, $*PERL.compiler.version ]
<camelia> rakudo-moar c201a7: OUTPUT«[v2016.07.24.g.31.eccd.7,
v2016.07.1.243.gc.201.a.76]␤»
<Zoffix> Oh. I guess camelia doesn't build every commit right away.

I did not create camelia, so was unfamiliar with her workings, but there was another bot whom I did create and knew for sure it was supposed to update to HEAD every hour... it didn't:

<Zoffix> s: dd $*VM.version, $*PERL.compiler.version
<SourceBaby> Zoffix, Something's wrong: ␤ERR: v2016.07.24.g.31.eccd.7
v2016​.07.1.243.gc.201.a.76␤Cannot resolve caller sourcery(Nil); none of
these signatures match:␤ ($thing, Str:D $method, Capture $c)␤
($thing, Str:D $method)␤ (&code)␤ (&code, Capture $c)␤
in block at -e line 6␤␤
* Zoffix gets a bad feeling

The Fallout

I didn't have to wonder about what the hell was going on for too long, since shortly thereafter a user in #perl6 channel turned up, saying they can't build:

<cuonglm> Hi, anyone got trouble when building 2016.08 rakudo
release from source I got `error: pathspec '2016.08' did not match
any file(s) know to git. Sounds like git history was overriden

After a short conversation with the person, it became obvious that while cloning and building the repo (or using the release I released) may have worked fine, simply git pulling ran into the issue with the tag.

It was obvious there was an issue and it needed fixing. The question was: how. The first suggestions seemed scary:

<mst> you can always burn the tag with fire and replace it with a fixed one.
<mst> the advantages may outweigh the risks

We already had users with checkouts with a broken tag asking for help in the chat channel and since I've sent the release announcement email, there may have been many users experiencing such a checkout issue. I didn't know what to do, so the first thing I did was panic and attempt to find someone else to fix my mess:

<Zoffix> jnthn, [Coke], TimToady are you around? I made a booboo with the tag

A lot of the core devs are in Europe, so there's a bit of timezone conflict with me, and with YAPC::Europe happening, some of them were not even in the chat at the time. Left to my own devices, I joined #git on Freenode, and the kind folks in there reaffirmed that replacing the tag might be a bad idea. Victory... I didn't have to learn how to do that:

<Zoffix> What if I delete the tag and create a new one.
What happens to people who have cloned the branch with the incorrect tag?
<_ikke_> Zoffix: A good question. Tags are kind of special as
in they're not expecting to change (and git somewhat ignores changing tags from the remote)
<_ikke_> Zoffix: man git tag has a section about it ("On retagging")
<gitinfo> Zoffix: the git-tag manpage is available at http://jk.gs/git-tag.html
<Zoffix> OK, then I think I'll leave it as it is, and wait for
someone smarter than me to figure out a fix :P
<Zoffix> Haha "Just admit you screwed up, and use
a different name." Yup, I'll go with that :P
<Zoffix> Thanks for the help!
<_ikke_> YW ;-)

With tag mangling out of the question and users having issues, the next answer was to make an emergency point release with a corrected tag, so I proceeded to do so:

<Zoffix> OK. I'm gonna do a 2016.08.1 'cause I'm not 100% sure
just changing the tag will fix everything and won't introduce new problems.

Making another NQP release, with a correct tag, seemed to resolve the issue on my box, but I wasn't sure there wasn't any more fallout about to happen with the current Rakudo release, so I made a point release of the compiler as well, just to be safe, and to sync up versions with NQP (so folks don't think a 2016.08.01 Rakudo release is missing, when they see a 2016.08.1 NQP release).

Jumping into the release guides at mid-point proved challenging, and considering I've just done a release, so did trying to keep track of which steps I've already done. Tired. Embarrassed. And too sober for the occasion. I headed to the pub, knowing this is the last time I'll make a release like this.

Part II: I, Robot

If you've heard of The Joel Test, you may've noticed Perl 6's release process described above currently fails item #2:

2. Can you make a build in one step?

Painfully aware of the issue, I was thinking about improving the situation ever since the first time I stepped up to cut a release. If you follow the #perl6-dev channel, you may have even seen me show off a couple of prototypes. Such as my Rele6sr web app that lets you keep track of release-blocking bug tickets:

Or my Rele6sr bot that reminds you about an upcoming release, giving a list of new tickets since last release, as well as the ChangeLog entries to review:

<Zoffix> Rele6sr: reminder
<Rele6sr> 🎺🎺🎺 Friends! I bear good news! Rakudo's release will
happen in just 1 days! Please update the ChangeLog with anything you worked
on that should be known to our users. 🎺🎺🎺
<Rele6sr> 🎺🎺🎺 Here are still-open new RT tickets since last release:
http://bug.perl6.party/1471608216.html And here is the git log output for commits
since last release: http://bug.perl6.party/1471608219.html 🎺🎺🎺

Today's incident indicated to me that it's time to stop messing around with prototypes and put out a complete plan of action.

Spread It Out

There are two things the Release Guide contains that may make it seem like automating releases is a tough nut to crack: reviewing bug tickets for blockers and populating the ChangeLog with needed items. We can't make a robot reliably do those things (yet?), so the largest part of the release process requests a human to do so. But... what if we spread that job throughout the entire month between the releases?

This month, I will create a web app that will keep track of all the reported tickets and will let the release manager log in and mark the tickets they reviewed as either release blockers or non-blockers. As a bonus, the prototype buggable bug-queue interfacing bot that some of you may have seen will also use the app to do its thing more correctly, and the app will also serve as a nicer interface to tickets for people who don't like our original RT website.

The same principle will apply to ChangeLog entries too, where the site will keep track of the commits the release manager has reviewed and added to the ChangeLog. As such, by "keeping state" around, the web app will let the release manager spend only a couple of minutes every few days reviewing changes and bugs, instead of cramming all of the work into a single session on the release day.

Build In One Step

Providing the release manager keeps up with the above steps throughout the month, on the release day they will have a single thing to do: issue the release command to the Rele6sr bot:

<Zoffix> Rele6sr: release
<Rele6sr> Yey! It's that time! Starting the release.
You can watch the progress on http://bug.perl6.party/some-url
Tell me to abort at any time to abort the process.

Since the issue review process has been subtracted from the release instructions, all that's left is grunt work that can be entirely automated. The bot will use Google Compute Engine API to fire up my 24-core dev box. It will then ssh into it and clone the repos, update versions, run the builds, run the tests, tag stuff, generate tarballs, test the tarballs, scp the tarballs to Rakudo's website, email the release announcement, and update the Wikipedia page (if there's an API for that). The human can watch all of that happen on a page that feeds a websocket with the output from Proc::Async and abort if anything goes wrong. The bot will automatically abort if any of the tests in the test suite fail. As a bonus, I'm hoping to make the bot fire up instances of VMs with several different OSes, to add an extra layer of testing in different environments.

Granted, at least with the first several releases, there will be safety stop-measures and verification steps where a human can ensure the robot is doing the right thing, and abort the process, if needed. But there's no technical limitation to the bot correctly cutting a release after being issued a single command.

Exterminate All Humans

Automating complex build jobs eliminates mistakes. A robot doesn't care if it has to rebuild from scratch a hundred times while you're trying to debug a release-blocker (OK, I think it doesn't care... any robots' rights activists in the audience to tell me I'm wrong?).

Both my today's mistake and the difficulty of fixing it could've been avoided if the release process were entirely automated. The fewer things you have to do manually, the fewer places a mistake can creep into.

The Joel Test, while an amusing read, has a lot of truth and simplicity to it. I suggest you re-evaluate any of the points your project fails the test on, and I believe you'll save yourself a lot of headache if you do so.

Conclusion

Today's events were a great learning experience and a catalyst for improvement. Humans are squishy and fallable and they make mistakes, especially out of habit.

Trying to follow a regular list of instructions during an irregular situation can prove difficult and introduce more mistakes, making a bad situation worse.

If you haven't yet, try to automate your release process as much as you can. I know Perl 6 will follow that advice. This was the last time I cut a release. My job was taken by a robot.

If you're following tech stuff, you probably know by now about the folks at Rust land working on some totally awesome error reporting capabilities. Since Perl 6 is also known for its Awesome Errors, mst inquired for some examples to show off to the rustaceans, and unfortunately I drew a blank...

Errors are something I try to avoid and rarely read in full. So I figured I'll hunt down some cool examples of Awesome Errors and write about them. While I could just bash my head on the keyboard and paste the output, that'd be quite boring to read, so I'll talk about some of the tricky errors that might not be obvious to beginners, and how to fix them.

Let the head bashing begin!

The Basics

Here's some code with an error in it;

say "Hello world!;
say "Local time is {DateTime.now}";

# ===SORRY!=== Error while compiling /home/zoffix/test.p6
# Two terms in a row (runaway multi-line "" quote starting at line 1 maybe?)
# at /home/zoffix/test.p6:2
# ------> say "⏏Local time is {DateTime.now}";
#     expecting any of:
#         infix
#         infix stopper
#         postfix
#         statement end
#         statement modifier
#         statement modifier loop

The first line is missing the closing quote on the string, so everything until the opening quote on the second line is still considered part of the string. Once the supposedly closing quote is found, Perl 6 sees word "Local," which it identifies as a term. Since two terms in a row are not allowed in Perl 6, the compiler throws an error, offering some suggestions on what it was expecting, and it detects we're in a string and suggests we check we didn't forget a closing quote on line 1.

The ===SORRY!=== part doesn't mean you're running the Canadian version of the compiler, but rather that the error is a compile-time (as compared to a run-time) error.

Nom-nom-nom-nom

Here's an amusing error. We have a subroutine that returns things, so we call it and use a for loop to iterate over the values:

sub things { 1 … ∞ }

for things {
    say "Current stuff is $_";
}

# ===SORRY!===
# Function 'things' needs parens to avoid gobbling block
# at /home/zoffix/test.p6:5
# ------> }⏏<EOL>
# Missing block (apparently claimed by 'things')
# at /home/zoffix/test.p6:5
# ------> }⏏<EOL>

Perl 6 lets you omit parentheses when calling subroutines. The error talks about gobbling blocks. What happens is the block we were hoping to give to the for loop is actually being passed to the subroutine as an argument instead. The second error in the output corroborates by saying the for loop is missing its block (and makes a suggestion it was taken by the our things subroutine).

The first error tells us how to fix the issue: Function 'things' needs parens, so our loop needs to be:

for things() {
    say "Current stuff is $_";
}

However, were our subroutine actually expecting a block to be passed, no parentheses would be necessary. Two code blocks side by side would result in "two terms in a row" error we've seen above, so Perl 6 knows to pass the first block to the subroutine and use the second block as the body of the for loop:

sub things (&code) { code }

for things { 1 … ∞ } {
    say "Current stuff is $_";
}

Did You Mean Levestein?

Here's a cool feature that will not only tell you you're wrong, but also point out what you might've meant:

sub levenshtein {}
levestein;

# ===SORRY!=== Error while compiling /home/zoffix/test.p6
# Undeclared routine:
#     levestein used at line 2. Did you mean 'levenshtein'?

When Perl 6 encounters names it doesn't recognize it computes Levenshtein distance for the things it does know to try to offer a useful suggestion. In the instance above it encountered an invocation of a subroutine it didn't know about. It noticed we do have a similar subroutine, so it offered it as an alternative. No more staring at the screen, trying to figure out where you made the typo!

The feature doesn't consider everything under the moon each time it's triggered, however. Were we to capitalize the sub's name to Levenshtein, we would no longer get the suggestion, because for things that start with a capital letter, the compiler figures it's likely a type name and not a subroutine name, so it checks for those instead:

class Levenshtein {}
Lvnshtein.new;

# ===SORRY!=== Error while compiling /home/zoffix/test.p6
# Undeclared name:
#    Lvnshtein used at line 2. Did you mean 'Levenshtein'?

Once You Go Seq, You Never Go Back

Let's say you make a short sequence of Fibonacci numbers. You print it and then you'd like to print it again, but this time square each member. What happens?

my $seq = (1, 1, * + * … * > 100);
$seq             .join(', ').say;
$seq.map({ $_² }).join(', ').say;

# 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144
# This Seq has already been iterated, and its values consumed
# (you might solve this by adding .cache on usages of the Seq, or
# by assigning the Seq into an array)
#   in block <unit> at test.p6 line 3

Ouch! A run-time error. What's happening is the Seq type we get from the the sequence operator doesn't keep stuff around. When you iterate over it, each time it gives you a value, it discards it, so once you've iterated over the entire Seq, you're done.

Above, we're attempting to iterate over it again, and so the Rakudo runtime cries and complains, because it can't do it. The error message does offer two possible solutions. We can either use the .cache method to obtain a List we'll iterate over:

my $seq = (1, 1, * + * … * > 100).cache;
$seq             .join(', ').say;
$seq.map({ $_² }).join(', ').say;

Or we can use an Array from the get go:

my @seq = 1, 1, * + * … * > 100;
@seq             .join(', ').say;
@seq.map({ $_² }).join(', ').say;

And even though we're storing the Seq in an Array, it won't get reified until it's actually needed:

my @a = 1 … ∞;
say @a[^10];

# OUTPUT:
# (1 2 3 4 5 6 7 8 9 10)

These Aren't The Attributes You're Looking For

Imagine you have a class. In it, you have some private attributes and you've got a method that does a regex match using the value of that attribute as part of it:

class {
    has $!prefix = 'foo';
    method has-prefix ($text) {
        so $text ~~ /^ $!prefix/;
    }
}.new.has-prefix('foobar').say;

# ===SORRY!=== Error while compiling /home/zoffix/test.p6
# Attribute $!prefix not available inside of a regex, since regexes are methods on Cursor.
# Consider storing the attribute in a lexical, and using that in the regex.
# at /home/zoffix/test.p6:4
# ------>         so $text ~~ /^ $!prefix⏏/;
#     expecting any of:
#         infix stopper

Oops! What happened?

It's useful to understand that as far as the parser is concerned, Perl 6 is actually braided from several languages: Perl 6, Quote, and Regex languages are parts of that braiding. This is why stuff like this Just Works™:

say "foo { "bar" ~ "meow" } ber ";

# OUTPUT:
# foo barmeow ber

Despite the interpolated code block using the same " quotes to delimit the strings within it as the quotes on our original string, there's no conflict. However, the same mechanism presents us with a limitation in regexes, because in them, the looked up attributes belong to the Cursor object responsible for the regex.

To avoid the error, simply use a temporary variable to store the $!prefix in—as suggested by the error message—or use the given block:

class {
    has $!prefix = 'foo';
    method has-prefix ($text) {
        given $!prefix { so $text ~~ /^ $_/ }
    }
}.new.has-prefix('foobar').say;

De-Ranged

Ever tried to access an element of a list that's out of range?

my @a = <foo bar ber>;
say @a[*-42];

# Effective index out of range. Is: -39, should be in 0..Inf
#  in block <unit> at test.p6 line 2

In Perl 6, to index an item from the end of a list, you use funky syntax: [*-42]. That's actually a closure that takes an argument (which is the number of elements in the list), subtracts 42 from it, and the return value is used as an actual index. You could use @a[sub ($total) { $total - 42 }] instead, if you were particularly bored.

In the error above, that index ends up being 3 - 42, or -39, which is the value we see in the error message. Since indexes cannot be negative, we receive the error, which also tells us the index must be 0 to positive infinity (with any indexes above what the list contains returning Any when looked up).

A Rose By Any Other Name, Would Code As Sweet

If you're an active user of Perl 6's sister language, the Perl 5, you may sometimes write Perl-5-isms in your Perl 6 code:

say "foo" . "bar";

# ===SORRY!=== Error while compiling /home/zoffix/test.p6
# Unsupported use of . to concatenate strings; in Perl 6 please use ~
# at /home/zoffix/test.p6:1
# ------> say "foo" .⏏ "bar";

Above, we're attempting to use Perl 5's concatenation operator to concatenate two strings. The error mechanism is smart enough to detect such usage and to recommend the use of the correct ~ operator instead.

This is not the only case of such detection. There are many. Here's another example, detecting accidental use of Perl 5's diamond operator, with several suggestions of what the programmer may have meant:

while <> {}

# ===SORRY!=== Error while compiling /home/zoffix/test.p6
# Unsupported use of <>; in Perl 6 please use lines() to read input, ('') to
# represent a null string or () to represent an empty list
# at /home/zoffix/test.p6:1
# ------> while <⏏> {}

Heredoc, Theredoc, Everywheredoc

Here's an evil error and there's nothing awesome about it, but I figured I'd mention it, since it's easy to debug if you know about it, and quite annoying if you don't. The error is evil enough that it may have been already improved if you're reading this far enough in the future from when I'm writing this.

Try to spot what the problem is... read the error at the bottom first, as if you were the one who wrote (and so, are familiar with) the code:

my $stuff = qq:to/END/;
Blah blah blah
END;

for ^10 {
    say 'things';
}

for ^20 {
    say 'moar things';
}

sub foo ($wtf) {
    say 'oh my!';
}

# ===SORRY!=== Error while compiling /home/zoffix/test.p6
# Variable '$wtf' is not declared
# at /home/zoffix/test.p6:13
# ------> sub foo (⏏$wtf) {

Huh? It's crying about an undeclared variable, but it's pointing to a signature of a subroutine. Of course it won't be declared. What sort of e-Crack is the compiler smoking?

For those who didn't spot the problem: it's the spurious semicolon after the closing END of the heredoc. The heredoc ends where the closing delimiter appears on a line all by itself. As far as the compiler is concerned, we've not seen the delimiter at END;, so it continues parsing as if it were still parsing the heredoc. A qq heredoc lets you interpolate variables, so when the parser gets to the $wtf variable in the signature, it has no idea it's in a signature of an actual code and not just some random text, so it cries about the variable being undeclared.

Won't Someone Think of The Reader?

Here's a great error that prevents you from writing horrid code:

my $a;
sub {
    $a.say;
    $^a.say;
}

# ===SORRY!=== Error while compiling /home/zoffix/test.p6
# $a has already been used as a non-placeholder in the surrounding sub block,
#   so you will confuse the reader if you suddenly declare $^a here
# at /home/zoffix/test.p6:4
# ------>         $^a⏏.say;

Here's a bit of a background: you can use the $^ twigil on variables to create an implicit signature. To make it possible to use such variables in nested blocks, this syntax actually creates the same variable without the twigil, so $^a and $a are the same thing, and the signature of the sub above is ($a).

In our code, we also have an $a in outer scope and supposedly we print it first, before using the $^ twigil to create another $a in the same scope, but one that contains the argument to the sub... complete brain-screw! To avoid this, just rename your variables to something that doesn't clash. How about some Thai?

my $ความสงบ = 'peace';
sub {
    $ความสงบ.say;
    $^กับตัวแปรของคุณ.say;
}('to your variables');

# OUTPUT:
# peace
# to your variables

Well, Colour Me Errpressed!

If your terminal supports it, the compiler will emit ANSI codes to colourize the output a bit:

for ^5 {
    say meow";
}

That's all nice and spiffy, but if you're, say, capturing output from the compiler to display it elsewhere, you may get the ANSI code as is, like 31m===[0mSORRY![31m===[0m.

That's awful, but luckily, it's easy to disable the colours: just set RAKUDO_ERROR_COLOR environmental variable to 0:

You can set it from within the program too. You just have to do it early enough, so put it somewhere at the start of the program and use the BEGIN phaser to set it as soon as the assignment is compiled:

BEGIN %*ENV<RAKUDO_ERROR_COLOR> = 0;

for ^5 {
    say meow";
}

An Exceptional Failure

Perl 6 has a special exception—Failure—that doesn't explode until you try to use it as a value, and you can even defuse it entirely by using it in boolean context. You can produce your own Failures by calling the fail subroutine and Perl 6 uses them in core whenever it can.

Here's a piece of code where we define a prefix operator for calculating the circumference of an object, given its radius. If the radius is negative, it calls fail, returning a Failure object:

sub prefix:<⟳> (\𝐫) {
    𝐫 < 0 and fail 'Your object warps the Universe a new one';
    τ × 𝐫;
}

say 'Calculating the circumference of the mystery object';
my $cₘ = ⟳ −𝑒;

say 'Calculating the circumference of the Earth';
my $cₑ = ⟳ 6.3781 × 10⁶;

say 'Calculating the circumference of the Sun';
my $cₛ = ⟳ 6.957 × 10⁸;

say "The circumference of the largest object is {max $cₘ, $cₑ, $cₛ} metres";

# OUTPUT:
# Calculating the circumference of the mystery object
# Calculating the circumference of the Earth
# Calculating the circumference of the Sun
# Your object warps the Universe a new one
#   in sub prefix:<⟳> at test.p6 line 2
#   in block <unit> at test.p6 line 7
#
# Actually thrown at:
#   in block <unit> at test.p6 line 15

We're calculating the circumference for a negative radius on line 7, so if it were just a regular exception, our code would die there and then. Instead, by the output we can see that we continue to calculate the circumference of the Earth and the Sun, until we get to the last line.

There we try to use the Failure in $cₘ variable as one of the arguments to the max routine. Since we're asking for the actual value, the Failure explodes and gives us a nice backtrace. The error message includes the point where our Failure blew up (line 15), where we received it (line 7) as well as where it came from (line 2). Pretty sweet!

Conclusion

Useful, descriptive errors are becoming the industry standard and Perl 6 and Rust languages are leading that effort. The errors must go beyond merely telling you the line number to look at. They should point to a piece of code you wrote. They should make a guess at what you meant. They should be referencing your code, even if they originate in some third party library you're using.

Most of Perl 6 errors display the piece of code containing the error. They use algorithms to offer valid suggestions when you mistyped a subroutine name. If you're used to other languages, Perl 6 will detect your "accent," and offer the correct way to pronounce your code in Perl 6. And instead of immediately blowing up, Perl 6 offers a mechanism to propagate errors right to the code the programmer is writing.

Perl 6 has Awesome Errors.

Read this article on Perl6.Party

Imagine you were playing with Perl 6 and you came across a buglet or you were having some fun with the Perl 6 bug queue—you'd like to debug a particular core subroutine or method, so where's the source for it at?

Asked such a question, you might be told it's in Rakudo compiler's GitHub repository. Depending on how deep down the rabbit hole you wish to go, you may also stop by NQP's repo, which is a subset of Perl 6 that's used in Rakudo, or the MoarVM's repo, which is the leading virtual machine Perl 6 runs on.

The answer is fine, but we can do better. We'd like to know exactly where da sauce is.

Stick to The Basics

The most obvious way is to just use grep command in the source repository. The code is likely in src/ directory, or src/core more specifically.

We'll use a regex that catches sub, method, and multi keywords. For example, here's our search for path sub or method:

$ grep -nER '^\s*(multi|sub|method|multi sub|multi method)\s+path' src/core

src/core/Cool.pm:229:    method path() { self.Stringy.IO }
src/core/CompUnit/Repository/Locally.pm:26:    method path-spec(CompUnit::Repository::Locally:D:) {
src/core/CompUnit/Repository/AbsolutePath.pm:46:    method path-spec() {
src/core/CompUnit/Repository/NQP.pm:32:    method path-spec() {
src/core/CompUnit/Repository/Perl5.pm:46:    method path-spec() {
src/core/CompUnit/PrecompilationStore/File.pm:93:    method path(CompUnit::PrecompilationId $compiler-id,
src/core/CompUnit/PrecompilationUnit.pm:17:    method path(--> IO::Path) { ... }
src/core/IO/Spec/Win32.pm:58:    method path {
src/core/IO/Spec/Unix.pm:61:    method path {
src/core/IO/Handle.pm:714:    method path(IO::Handle:D:)            { $!path.IO }

It's not too terrible, but it's a rather blunt tool. We have these problems:

• There are false positives; we have several path-spec methods found
• It doesn't tell us which of the results is for the actual method we have in our code. There's Cool, IO::Spec::Unix, and IO::Handle all with method path in them. If I call "foo".IO.path, which of those get called?

The last one is particularly irksome, but luckily Perl 6 can tell us where the source is from. Let's ask it!

But here's line number... So code me maybe

The Code class from which all subs and methods inherit provides .file and .line methods that tell which file that particular Code is defined in, including the line number:

say "The code is in {.file} on line {.line}" given &foo;

sub foo {
    say 'Hello world!';
}

# OUTPUT:
# The code is in test.p6 on line 3

That looks nice and simple, but it gets more awkward with methods:

class Kitty {
    method meow {
        say 'Meow world!';
    }
}

say "The code is in {.file} on line {.line}" given Kitty.^can('meow')[0];

# OUTPUT:
# The code is in test.p6 on line 2

We got extra cruft of the .^can metamodel call, which returns a list of Method objects. Above we use the first one to get the .file and .line number from, but is it really the method we were looking for? Take a look at this example:

class Cuddly {
    method meow ('meow', 'meow') {
        say 'Meow meow meow!';
    }
}

class Kitty is Cuddly {
    multi method meow ('world') {
        say 'Meow world!';
    }

    multi method meow ('meow') {
        say 'Meow meow';
    }
}

We have a method meow in one class and in another class we have two multi methods meow. How can we print the location of the last method, the one that takes a single 'meow' as an argument?

First, let's take a gander at all the items .^can returns:

say Kitty.^can('meow');
# OUTPUT:
# (meow meow)

Wait a minute, we have three methods in our code, so how come we only have two meows in the output? Let's print the .file and .line for both meows:

for 0, 1 {
    say "The code is in {.file} on line {.line}"
        given Kitty.^can('meow')[$_];
}
# OUTPUT:
# The code is in gen/moar/m-CORE.setting on line 587
# The code is in test.p6 on line 2

The second meow gives us a sane result; it's our method defined in class Cuddly. The first one, however, gives us some weird file.

What's happening here is the line is referencing the proto for the multies. Since in this case instead of providing our own proto we use the autogenerated one, the referenced file has nothing to do with our code. We can, of course, add a proto into the code, but then the line number would still reference the proto, not the last meow method. Is there anything that we can do?

You .cando It!

The Routine class, from which both Method and Sub classes inherit, provides the .cando method. Given a Capture, it returns a list of candidates that can handle it, with the narrowest candidate first in the list, and since the returned object is a Code, we can query its specific .file and .line:

class Cuddly {
    method meow ('meow', 'meow') {
        say 'Meow meow meow!';
    }
}

class Kitty is Cuddly {
    multi method meow ('world') {
        say 'Meow world!';
    }

    multi method meow ('meow') {
        say 'Meow meow';
    }
}

my $code = gather {
    for Kitty.^can('meow') -> $meth {
        .take for $meth.cando: \(Kitty, 'meow');
    }
}

say "The code is in {.file} on line {.line}" with $code[0];

# OUTPUT:
# The code is in test.p6 on line 12

Hooray! We got the correct location of the multi we wanted. We still have our two classes with three meow methods total. On line 17–21 we loop over the two meow Methods the .^can metamodel call gives us. For each of them we call the .cando method with the Capture that matches the multi we want (note that we do need to provide the needed object as the first argument of the Capture). We then .take all found candidates to gather them into the $code variable.

The first value we get is the narrowest candidate and is good 'nuf for us, so we call the .file and .line on it, which gives us the location we were looking for. Sounds like we nailed this .file and .line business down rather well. Let's dive into the core, shall we?

Can't see the core files for the setting

If this is the first time you're to see the print out of the .file/.line for some core stuff, you're in for a surprise. Actually, we've already seen the surprise, but you may have thought it to be a fluke:

say "{.file}:{.line}" given &say;
# OUTPUT:
# gen/moar/m-CORE.setting:29038

All of the nice, good looking files you see in src/core in the repo actually get compiled into one giant file called the "setting." My current setting is 40,952 lines long and the .line of core subs and methods refers to one of those thousands of lines.

Now sure, we could pop the setting open and watch our editor grind to a stuttering halt (I'm looking at you, Atom!). However, that doesn't help us find the right repo file to edit if we want to make changes to how it works. So what do we do?

A keen eye will look at the contents of the setting or at the file that generates it and notice that for each of the separate files in the repo, the setting has this type of comment before the contents of the file are inserted into the setting:

#line 1 src/core/core_prologue.pm

This means if we're clever enough, we can write a sub that translates a line number in the setting to the separate file we can locate in the repo. Here's a plan of action: we pop open the setting file and read it line by line. When we encounter one of the above comments, we make a note of which file we're in as well as how many lines deep in the setting we're currently at.

The location of the setting file may differ, depending on how you installed Perl 6, but on my system (I use rakudobrew), it's in $*EXECUTABLE.parent.parent.parent.child('gen/moar/m-CORE.setting'), so the code for finding the actual file that defines our core sub or method is this:

sub real-location-for ($wanted) {
    state $setting = $*EXECUTABLE.parent.parent.parent.child: 'gen/moar/m-CORE.setting';
    my ($cur-line-num, $offset) = 0, 0;
    my $file;
    for $setting.IO.lines -> $line {
        return %( :$file, :line($cur-line-num - $offset), )
            if ++$cur-line-num == $wanted;

        if $line ~~ /^ '#line 1 ' $<file>=\S+/ {
            $file   = $<file>;
            $offset = $cur-line-num + 1;
        }
    };
    fail 'Were not able to find location in setting.';
}

say "{.<file>}:{.<line>}" given real-location-for &say.line;


# OUTPUT:
# src/core/io_operators.pm:17

The $wanted contains the setting line number given to us by .line call and the $cur-line-num contains the number of the current line we're examining. We loop until the $cur-line-num reaches $wanted and return a Hash with the results. For each line that matches our special comment, we store the real name of the file the code is from into $file and store the $offset of the first line of the code in that file. Once done, we simply subtract the $offset from the setting $cur-line-num and we get the line number in the source file.

This is pretty awesome and useful, but it's still not what I had in mind when I said we wanted to know exactly where da sauce is. I don't want to clone the repo and go to the repo and open my editor. I want to just look at code.

If it's worth doing, it's worth overdoing

There's one place where we can stare at Rakudo's source code until it blushes and looks away: GitHub. Since our handy sub gives us a filename and a line number, we can construct a URL that points to a specific file and line in the source code, like this one, for example: https://github.com/rakudo/rakudo/blob/nom/src/core/Str.pm#L16

There's an obvious problem with such an approach: the URL points to the master branch (called nom, for "New Object Model," in Rakudo). Commits go into the repo daily, and unless we rebuild our Perl 6 several times a day, there's a good chance the location our GitHub URL points to is wrong.

Not only do we have to point to a specific file and line number, we have to point to the right commit too. On GitHub's end, it's easy: we just replace nom in the URL with the appropriate commit number—we just need Rakudo to tell us what that number is.

The two dynamic variables $*VM and $*PERL contain some juicy information. By introspecting them, we can locate some useful info and what looks like commit prefix parts in version numbers:

say $*VM.^methods;
# (BUILD platform-library-name Str gist config prefix precomp-ext
# precomp-target precomp-dir name auth version signature desc)

say $*VM.version;
# v2016.06

say $*PERL.^methods;
# (BUILD VMnames DISTROnames KERNELnames Str gist compiler name auth version
# signature desc)

say $*PERL.compiler.^methods;
# (BUILD build-date Str gist id release codename name auth version
# signature desc)

say $*PERL.compiler.version;
# v2016.06.10.g.7.cff.429

Rakudo is a compiler and so we're interested in the value of $*PERL.compiler.version. It contains the major release version, followed by g, followed by the commit prefix of this particular build. The prefix is split up on number-letter boundaries, so we'll need to join up all the bits and split on g. But, take a look at $*VM.version, which is the version of the virtual machine we're running the code on. There aren't any gs and commits in it and for a good reason: it's a tagged major release, and the name of the tag is the version. The same will occur for Rakudo on release builds, like the ones shipped with Rakudo Star. So we'll need to check for such edge cases and this is the code:

my $where = .Str ~~ /g/
    ?? .parts.join.split("g")[*-1]
    !! .Str
given $*PERL.compiler.version;

given a $*PERL .compiler .version, if it contains letter g, join up version bits, split on g, and the last portion will be our commit prefix; if it doesn't contain letter g, then we're dealing with a release tag, so we'll take it as-is. All said and done, our code for locating source becomes this:

my $where = .Str ~~ /g/
    ?? .parts.join.split("g")[*-1]
    !! .Str
given $*PERL.compiler.version;

say [~] 'https://github.com/rakudo/rakudo/blob/',
        $where, '/', .<file>, '#L', .<line>
given real-location-for &say.line;

# OUTPUT:
# https://github.com/rakudo/rakudo/blob/c843682/src/core/io_operators.pm#L17

Hey! Awesome! We got a link that points to the correct commit and file! Let celebrations begin! Wait. What? You followed the link and noticed the line number is not quite right? What gives? Did we mess up our algorithm?

Crank Up The Insanity

If you take a look again at the script that generates the setting file, you'll notice it strips things: comments and special backend-specific chunks of code.

There are two ways to fix this. The sane approach would be to commit a change that would make that script insert an empty line for each line it skips and then pretend that we didn't commit that just to make our personal project work. Then, there's the Zoffix Way to fix this: we got the GitHub link, so why don't we fetch that code and figure out what the right line number is. Hey! That second way sounds much more fun! Let's do just that!

The one link we've seen so far is this: https://github.com/rakudo/rakudo/blob/c843682/src/core/iooperators.pm#L17. It's not quite what we want, since it's got HTML and bells and whistles in it. We want raw code and GitHub does offer that at a slightly different URL: https://raw.githubusercontent.com/rakudo/rakudo/c843682/src/core/iooperators.pm. The plan of action then becomes:

• Get the line number in the setting
• Use our real-location-for sub to get the filename and sorta-right line number in a source file
• Get the commit our compiler was built with
• Generate a GitHub URL for raw code for that file on that commit and fetch that code
• Use the same algorithm as in the setting generating script to convert the code we fetched into the version that lives in our setting, while keeping track of the number of lines we strip
• When we reach the correct line number in the converted file, we adjust the original line number we had by the number of lines we stripped
• Generate a regular GitHub URL to the commit, file, and corrected line number
• ???
• Profit!

I could go over the code, but it's just a dumb, unfun algorithm, and most importantly, you don't need to know it. Because... there's a module that does just that!

What Sorcery Is This?

The module is called CoreHackers::Sourcery and when you use it, it'll augment the Code class and all core classes that inherit from it with .sourcery method, as well as provide a sourcery subroutine.

So, to get the location of the code for say sub, just run:

use CoreHackers::Sourcery;
&say.sourcery.put;

# OUTPUT:
# src/core/io_operators.pm:20 https://github.com/rakudo/rakudo/blob/c843682/src/core/io_operators.pm#L20

That gives us the correct location of the proto. We can either pop open a file in a repo checkout or view the code at the provided GitHub URL.

Want to get the location of a specific multi? There's no need to mess with .cando! The arguments you give to the .sourcery method will be used to select the best matching multi, so to find the location of the say multi that will handle say "foo" call, just run:

&say.sourcery("foo").put;

# OUTPUT:
# src/core/io_operators.pm:22 https://github.com/rakudo/rakudo/blob/c843682/src/core/io_operators.pm#L22

That covers the subs. For methods, you can go with the whole .^can meta dance, but we like simple things, and so we'll use the subroutine form of sourcery:

put sourcery Int, 'abs';         # method of a type object
put sourcery 42,  'split';       # method of an Int object
put sourcery 42,  'base', \(16); # best candidate for `base` method called with 16 as arg

This is pretty handy. And the whole hitting the GitHub thing? The module will cache the code fetched from GitHub, so things like this won't take forever:

put "Int.{.name} is at {.sourcery}" for Int.^methods;

However, if you do actually run that code, after some output you'll be greeted with this error:

# Method 'sourcery' not found for invocant of class 'Method+{Callable[Bool:D]}'
#   in block  at test.p6 line 1
#   in block <unit> at test.p6 line 1

The class it mentions is not a pure Method object, but has a mixin in it. While CoreHackers::Sourcery recomposes all core subclasses of Code class after augmenting it, it doesn't do that for such mixes, so you'd have to recompose them yourself:

for Int.^methods {
    .WHAT.^compose;
    put "Int.{.name} is at {.sourcery}" ;
}

Or better still, just use the subroutine form of sourcery:

put "Int.{.name} is at {sourcery $_}" for Int.^methods;

Do It For Me

For most stuff, we wouldn't want to do a whole bunch of typing to use a module and call subs and then copy/paste URLs or filenames. You'll notice sourcery returns a list of two items: the filename and the URL. This means we can make some nice and short aliases to call it and automatically pop open either our editor or web browser:

$ alias sourcery='perl6 -MCoreHackers::Sourcery -MMONKEY-SEE-NO-EVAL \
    -e '\''run "atom", "/home/zoffix/rakudo/" \
        ~ EVAL "sourcery(@*ARGS[0])[0]" '\'''

$ alias sourcery-web='perl6 -MCoreHackers::Sourcery -MMONKEY-SEE-NO-EVAL \
    -e '\''run "firefox", EVAL "sourcery(@*ARGS[0])[1]" '\'''

# opens Atom editor at the spot to edit code for Int.base
$  sourcery 'Int, "base"'

# opens Firefox, showing code for Int.base
$  sourcery 'Int, "base"'

We EVAL the argument we give to these aliases, so be careful with them. For sourcery alias, we run the Atom editor and give it the file to open. I prepended the location of my local Rakudo checkout, but you'd use yours. Most editors support opening file:line-number format to open files at a particular spot; if yours doesn't, modify the command.

For sourcery-web we use the URL returned by sourcery and open Firefox browser at this location. And just like that, with a few keystrokes, we can jump in to view or edit the code for a particular core sub or method in Rakudo!

Conclusion

We've learned where Rakudo's source lives, how to find the commit the current compiler is built off, and how to locate the source code for a particular sub or method in a giant file called the setting. We then further hacked away the inconveniences by getting to the actual place in the source code we can edit, culminating with a shiny module and a couple of handy command line aliases.

Happy hacking!

UPDATE 2016.08.05

Inspired by this blog post, lizmat++ has changed the setting generation script to not skip any lines, so making adjustments to line numbers by fetching source from GitHub is no longer necessary, as the line numbers match up with the original source.

Read this article on Perl6.Party

While testing a fix for one of the Less Than Awesome behaviours in standalone Signature objects, I came across a bugglet. Smartmatching two Signatures throws, while spilling a bit of the guts:

<Zoffix> m: my $m = method ($a: $b) { }; say $m.signature ~~ :($a, $b);
<camelia> rakudo-moar 46838d: OUTPUT«Method 'type' not found for invocant of class 'Any'␤ in block at line 1␤␤»

So I figured I'll write about fixing it, 'cause hacking on internals is lots of fun. Let's roll!

Golf It Down

The less code there is to reproduces the bug, the fewer places there are for that bug to hide. We have a detached method and then we smartmatch its signature against something else. Let's try to golf it down a bit and smartmatch two Signatures, without involving a method:

<Zoffix> m: :($a, $b) ~~ :($a, $b);
<camelia> rakudo-moar 46838d: ( no output )

The bug disappeared, so perhaps out Signature on the left doesn't contain the stuff that triggers the bug. Let's dump the signature of the method to see what we should match against:

<Zoffix> m: my $m = method ($a: $b) { }; say $m.signature <camelia> rakudo-moar 46838d: OUTPUT«($a: $b, *%_)␤»

Aha! It has a slurpy hash: *%_. Let's try matching a Signature with a slurpy in it:

<Zoffix> m: :(*%) ~~ :();
<camelia> rakudo-moar 46838d: OUTPUT«Method 'type' not found for invocant of class 'Any'␤ in block at line 1␤␤»

And there we go: hole in three. Let's proceed.

Roast It

There's an official Perl 6 test suite that Rakudo must pass to be called a Perl 6 compiler. Since we got a bug on our hands, we should add a test for it to the test suite to ensure it doesn't rear its ugly head again.

The copy of the repo gets automatically cloned into t/spec when you run make spectest in Rakudo's checkout. If you don't have a commit bit, you can just change the remote/branch of that checkout to your fork:

cd t/spec
git remote rm origin
git remote add origin https://github.com/YOURUSERNAME/roast
git checkout your-branch
cd ../..

It may be tricky to figure out which file to put the test in, if you're new. You can always ask the good folks on irc.freenode.net/#perl6 for advice. In this case, I'll place the test into S06-signature/outside-subroutine.t

While not required, I find it helpful to open a ticket for the bug. This way I can reference it in my fix in the compiler repo, I can reference it in the commit to the test repo, and people get a place where to tell me why I'm being stupid when I am. I opened this bug as RT#128795.

Now, for the code of the test itself. I'll adjust the plan at the top of the file to include however many tests I'm writing—in this case one. I'll use the lives-ok test sub and stick our buggy golfed code into it. Here's the diff of the changes to the file; note the reference to the ticket number in the comment before the test:

@@ -1,7 +1,7 @@
  use v6;
  use Test;

 -plan 3;
 +plan 4;

  # RT #82946
  subtest 'signature binding outside of routine calls' => {
 @@ -25,4 +25,7 @@ subtest 'smartmatch on signatures with literal strings' => {
  # RT #128783
  lives-ok { EVAL ’:($:)‘ }, ’signature marker is allowed in bare signature‘;

 +# RT #128795
 +lives-ok { :(*%)~~ :() }, 'smartmatch with no slurpy on right side';
 +
  # vim: ft=perl6

Run the file now to ensure the test fails. Hint: some files have fudging; explaining it is out of the scope of this article, but if you notice failures you're not expecting, look it up.

$ make t/spec/S06-signature/outside-subroutine.t
...
Test Summary Report
-------------------
t/spec/S06-signature/outside-subroutine.t (Wstat: 256 Tests: 4 Failed: 1)
  Failed test:  4
  Non-zero exit status: 1

With the test in place, it's time to look at some source code. Let the bug hunt begin!

Make it Saucy

Our bug involves a Smartmatch operator, which aliases the left side to the topic variable $_ and calls .ACCEPTS method on the right side with it. Both of our sides are Signature objects, so let's pop open Rakudo's sauce code for that class.

In the Rakudo's repo, directory src/core/ contains most of the built in types in separate files named after those types, so we'll just pop open src/core/Signature.pm in the editor and locate the definition of method ACCEPTS.

There are actually four multis for ACCEPTS. Here's the full code. Don't try to understand all of it, just note its size.

``` multi method ACCEPTS(Signature:D: Capture $topic) { nqp::p6bool(nqp::p6isbindable(self, nqp::decont($topic))); }

multi method ACCEPTS(Signature:D: @topic) {
    self.ACCEPTS(@topic.Capture)
}

multi method ACCEPTS(Signature:D: %topic) {
    self.ACCEPTS(%topic.Capture)
}

multi method ACCEPTS(Signature:D: Signature:D $topic) {
    my $sclass = self.params.classify({.named});
    my $tclass = $topic.params.classify({.named});
    my @spos := $sclass{False} // ();
    my @tpos := $tclass{False} // ();

    while @spos {
        my $s;
        my $t;
        last unless @tpos && ($t = @tpos.shift);
        $s=@spos.shift;
        if $s.slurpy or $s.capture {
            @spos=();
            @tpos=();
            last;
        }
        if $t.slurpy or $t.capture {
            return False unless any(@spos) ~~ {.slurpy or .capture};
            @spos=();
            @tpos=();
            last;
        }
        if not $s.optional {
            return False if $t.optional
        }
        return False unless $t ~~ $s;
    }
    return False if @tpos;
    if @spos {
        return False unless @spos[0].optional or @spos[0].slurpy or @spos[0].capture;
    }

    for flat ($sclass{True} // ()).grep({!.optional and !.slurpy}) -> $this {
        my $other;
        return False unless $other=($tclass{True} // ()).grep(
            {!.optional and $_ ~~ $this });
        return False unless +$other == 1;
    }

    my $here=($sclass{True}:v).SetHash;
    my $hasslurpy=($sclass{True} // ()).grep({.slurpy});
    $here{@$hasslurpy} :delete;
    $hasslurpy .= Bool;
    for flat @($tclass{True} // ()) -> $other {
        my $this;

        if $other.slurpy {
            return False if any($here.keys) ~~ -> Any $_ { !(.type =:= Mu) };
            return $hasslurpy;
        }
        if $this=$here.keys.grep( -> $t { $other ~~ $t }) {
            $here{$this[0]} :delete;
        }
        else {
            return False unless $hasslurpy;
        }
    }
    return False unless self.returns =:= $topic.returns;
    True;
}

```

The error we get from the bug mentions .type method call and there is one such method call in the code above (close to the end of it). In this case, there's quite a bit of code to sort through. It would be nice to be able to play around with it, stick a couple of dd or say calls to dump out variables, right?

That approach, however, is somewhat annoying because after each change we have to recompile the entire Rakudo. On the meatiest box I got, it takes about 60 seconds. Not the end of the world, but there's a way to make things lightning fast!

Mix Your Fix

We need to fix a bug in a method of a class. Another way to think of it is: we need to replace a broken method with a working one. Signature class is just like any other class, so if we want to replace one of its methods, we can just mix in a role!

The broken ACCEPTS will continue to live in the compiler, and we'll pop open a separate playground file and define a role—let's calls it FixedSignature—in it. To get our new-and-improved ACCEPTS method in standalone signature objects, we'll use the but operator to mix the FixedSignature in.

Here's the role, the mixing in, and the code that triggers the bug. I'll leave out method bodies for brieviety, but there's they are the same as in the code above.

role FixedSignature {
    multi method ACCEPTS(Signature:D: Capture $topic)     { #`(redacted for brevity) }
    multi method ACCEPTS(Signature:D: @topic)             { #`(redacted for brevity) }
    multi method ACCEPTS(Signature:D: %topic)             { #`(redacted for brevity) }
    multi method ACCEPTS(Signature:D: Signature:D $topic) { #`(redacted for brevity) }
}

my $a = :(*%) but FixedSignature;
my $b = :()   but FixedSignature;

say $a ~~ $b;

There are two more things we need to do for our role to work properly. First, we're dealing with multis and right now the multis in our role are creating ambiguities with the multis in the original Signature class. To avoid that, we'll define a proto:

proto method ACCEPTS (|) { * }

Since the code is using some NQP, we also need to bring in those features into our playground file with the role. Just add the appropriate pragma at the top of the file:

use MONKEY-GUTS;

With these modifications, our final test file becomes the following:

use MONKEY-GUTS;

role FixedSignature {
    proto method ACCEPTS (|) { * }

    multi method ACCEPTS(Signature:D: Capture $topic)     { #`(redacted for brevity) }
    multi method ACCEPTS(Signature:D: @topic)             { #`(redacted for brevity) }
    multi method ACCEPTS(Signature:D: %topic)             { #`(redacted for brevity) }
    multi method ACCEPTS(Signature:D: Signature:D $topic) { #`(redacted for brevity) }
}

my $a = :(*%) but FixedSignature;
my $b = :()   but FixedSignature;

say $a ~~ $b;

And with this trick in place, we now have a rapid-fire weapon to hunt down the bug with—the changes we make compile instantly.

Pull The Trigger

Now, we can debug the code just like any other. I prefer applying liberal amounts of dd (or say) calls and dumping out the variables to ensure their contents match expectations.

The .type method call our error message mentions is in this line:

return False if any($here.keys) ~~ -> Any $_ { !(.type =:= Mu) };

It calls it on the keys of $here, so let's dump the $here before that statement:

...
dd $here
return False if any($here.keys) ~~ -> Any $_ { !(.type =:= Mu) };
...
# OUTPUT:
# SetHash $here = SetHash.new(Any)

Here's our offending Any, let's go up a bit and dump the $here right where it's defined:

...
my $here=$sclass{True}.SetHash;
dd $here;
...
# OUTPUT:
# SetHash $here = SetHash.new(Any)

It's still there, and for a good reason. If we trace the creation of $sclass, we'll see it's this:

my $sclass = self.params.classify({.named});

The params of the Signature on the right of the smartmatch get classified based on whether they are named or not. The named parameters will be inside a list under the True key of $sclass. Since we do not have any named params, there won't be such a key, and we can verify that with this bit of code:

:().params.classify(*.named).say
# OUTPUT:
# {}

When we go to define $here, we get an Any from $sclass{True}, since that key doesn't exist, and when we call .SetHash on it, we get our problematic Sethash object with an Any in it. And so, we have our fix for the bug: ensure the True key in $sclass is actually there before creating a SetHash out of its value:

my $here=($sclass{True}:v).SetHash;

Add that to our playground file with the FixedSignature role in it, run it, and verify the fix works. Now, simply transplant the fix back into src/core/Signature.pm and then compile the compiler.

perl Configure.pl --gen-moar --gen-nqp --backends=moar
make
make test
make install

Verify our fix worked before we proceed onto the final stages:

$ make t/spec/S06-signature/outside-subroutine.t
...
All tests successful.
Files=1, Tests=4,  1 wallclock secs ( 0.03 usr  0.00 sys +  0.32 cusr  0.02 csys =  0.37 CPU)
Result: PASS

A Clean Kill

So far, all we know is the bug we found was fixed and the tests we wrote for it pass. However, before we ship our fix, we must ensure we didn't break anything else. There are other devs working from the same repo and you'll be interfering with their work if you break stuff.

Run the full Roast test suite with make spectest command. You can use the TEST_JOBS environmental variable to specify the number of simultaneous tests. Generally a value slightly higher than the available cores works the fastest... and cores make all the difference. On my 24-core VM I cut releases on, the spectest completes in about 1 minute and 15 seconds. On my 2-core web server, it takes about 25 minutes. You get the idea.

TEST_JOBS=28 make spectest
...
All tests successful.
Files=1111, Tests=52510, 82 wallclock secs (13.09 usr 2.44 sys + 1517.34 cusr 97.67 csys = 1630.54 CPU)
Result: PASS

Once the spectest completes and we have the clean bill of health, we're ready to ship our fix. Commit the Rakudo fix, then go into t/spec and commit the Roast fix:

git commit -m 'Fix Smartmatch with two signatures, only one of which has slurpy hash' \
           -m 'Fixes RT#128795' src/core/Signature.pm
git push

cd t/spec
git commit -m 'smartmatch on signature with no slurpy on right side does not crash' \
           -m 'RT#128795' S06-signature/outside-subroutine.t
git push

If you're pushing to your fork of these projects, you have to go the extra step and submit a Pull Request (just go to your fork and GitHub should display a button just for that).

And we're done! Celebrate with the appropriate amount of fun.

Conclusion

Rakudo bugs can be easy to fix, requiring not much more than knowledge of Perl 6. To fix them, you don't need to re-compile the entire compiler, but can instead define a small role with a method you're trying to fix and modify and recompile just that.

It's important to add tests for the bug into the official test suite and it's also important to run the full spectest after you fix the bug. But most important of all, is to have fun fixing it.

-Ofun

Read this article on Perl6.Party

I wrote my first Perl 6 program—a New Years IRC Party bot—around Christmas, 2015. The work included releasing the IRC::Client module, and given my virginity with the language and blood alcohol level appropriate for the Holiday Season, the module ended up sufficiently craptastic.

Recently, I needed a tool for some Perl 6 bug queue work, so I decided to lock myself up for a weekend and re-design and re-write the module from scratch. Multiple people bugged me to do so over the past months, so I figured I'd also write a tutorial for how to use the module—as an apology for being a master procrastinator. And should IRC be of no interest to you, I hope the tutorial will prove useful as a general example of async, non-blocking interfaces in Perl 6.

The Basics

To create an IRC bot, instantiate an IRC::Client object, giving it some basic info, and call the .run method. Implement all of the functionality you need as classes with method names matching the events you want to listen to and hand those in via the .plugins attribute. When an IRC event occurs, it's passed to all of the plugins, in the order you specify them, stopping if a plugin claims it handled the event.

Here's a simple IRC bot that responds to being addressed in-channel, notices, and private messages sent to it. The response is the uppercased original message the bot received:

use IRC::Client;
.run with IRC::Client.new:
    :nick<MahBot>
    :host<irc.freenode.net>
    :channels<#perl6>
    :debug
    :plugins(class { method irc-to-me ($_) { .text.uc } })

And here's what the bot looks like when running:

<Zoffix> MahBot, I ♥ you!
<MahBot> Zoffix, I ♥ YOU!

The :nick, :host, and :channels are the nick for your bot, the server it should connect to, and channels it should join. The :debug controls how much debugging output to display. We'll set it to value 1 here, for sparse debug output, just to see what's happening. Tip: install the optional Terminal::ANSIColor module to make debug output purty:

For the .plugins attribute, we hand in an anonymous class. If you have multiple plugins, just shove them all in in the order you want them to receive events in:

:plugins(PlugFirst.new, PlugSecond.new(:conf), class { ... })

The plugin class of our uppercasing bot has a single method that listens to irc-to-me event, triggered whenever the bot is addressed in-channel or is sent a private message or notice. It receives a single argument: one of the objects that does the IRC::Client::Message role. We stick it into the $_ topical variable to save a bit of typing.

We reply to the event by returning a value from the method. The original text is contained inside the .text attribute of the message object, so we'll call .uc method on it to uppercase the content and that's what our reply will be.

As awesome as our uppercasing bot is, it's as useful as an air conditioner on a polar expedition. Let's teach it some tricks.

Getting Smarter

We'll call our new plugin Trickster and it'll respond to commands time—that will give the local time and date—and temp—that will convert temperature between Fahrenheit and Celsius. Here's the code:

use IRC::Client;

class Trickster {
    method irc-to-me ($_) {
        given .text {
            when /time/ { DateTime.now }
            when /temp \s+ $<temp>=\d+ $<unit>=[F|C]/ {
                when $<unit> eq 'F' { "That's {($<temp> - 32) × .5556}°C" }
                default             { "That's { $<temp> × 1.8 + 32   }°F" }
            }
            'huh?'
        }
    }
}

.run with IRC::Client.new:
    :nick<MahBot>
    :host<irc.freenode.net>
    :channels<#perl6>
    :debug
    :plugins(Trickster)

<Zoffix> MahBot, time
<MahBot> Zoffix, 2016-07-23T19:00:15.795551-04:00
<Zoffix> MahBot, temp 42F
<MahBot> Zoffix, That's 5.556°C
<Zoffix> MahBot, temp 42C
<MahBot> Zoffix, That's 107.6°F
<Zoffix> MahBot, I ♥ you!
<MahBot> Zoffix, huh?

The code is trivial: we pass the given text over a couple of regexes. If it contains word time, we return the current time. If it contains word temp we do the appropriate math, based on whether the given number is postfixed by an F or a C. And if no matches happen, we end up returning the inquisitive huh?.

There's an obvious problem with this new and improved plugin: the bot no longer loves me! And while I'll survive the heartache, I doubt any other plugin will teach the bot to love again, as Trickster consumes all irc-to-me events, even if it doesn't recognize any of the commands it can handle. Let's fix that!

Passing The Buck

There's a special value that can be returned by the event handler to signal that it did not handle the event and that it should be propagated to further plugins and event handlers. That value is provided by the .NEXT attribute offered by the IRC::Client::Plugin role, which a plugin does to obtain that attribute. The role is automatically exported when you use IRC::Client.

Let's look at some code utilizing that special value. Note that since .NEXT is an attribute and we can't look up attributes on type objects, you need to go the extra step and instantiate your plugin classes when giving them to :plugins.

use IRC::Client;

class Trickster does IRC::Client::Plugin {
    method irc-to-me ($_) {
        given .text {
            when /time/ { DateTime.now }
            when /temp \s+ $<temp>=\d+ $<unit>=[F|C]/ {
                when $<unit> eq 'F' { "That's {($<temp> - 32) × .5556}°C" }
                default             { "That's { $<temp> × 1.8 + 32   }°F" }
            }
            $.NEXT;
        }
    }
}

class BFF does IRC::Client::Plugin {
    method irc-to-me ($_) {
        when .text ~~ /'♥'/ { 'I ♥ YOU!' };
        $.NEXT;
    }
}

.run with IRC::Client.new:
    :nick<MahBot>
    :host<irc.freenode.net>
    :channels<#perl6>
    :debug
    :plugins(Trickster.new, BFF.new)

<Zoffix> MahBot, time
<MahBot> Zoffix, 2016-07-23T19:37:45.788272-04:00
<Zoffix> MahBot, temp 42F
<MahBot> Zoffix, That's 5.556°C
<Zoffix> MahBot, temp 42C
<MahBot> Zoffix, That's 107.6°F
<Zoffix> MahBot, I ♥ you!
<MahBot> Zoffix, I ♥ YOU!

We now have two plugins that both subscribe to irc-to-me event. The :plugins attribute receives Trickster plugin first, so its event handler will be run first. If the received text does not match either of the Trickster's regexes, it returns $.NEXT from the method.

That signals the Client Object to go hunting for other handlers, so it gets to BFF's irc-to-me handler. There, we reply if the input contains a heart, if not, we pre-emptively return $.NEXT here too.

While the bot got its sunny disposition back, it did so at the cost of quite a bit of extra typing. What can we do about that?

Multify All The Things!

Perl 6 supports multi-dispatch as well as type constraints in signatures. On top of that, smartmatch against IRC::Client's message objects that have a .text attribute uses the value of that attribute. Combine all three of those features and you end up with ridiculously concise code:

use IRC::Client;
class Trickster {
    multi method irc-to-me ($ where /time/) { DateTime.now }
    multi method irc-to-me ($ where /temp \s+ $<temp>=\d+ $<unit>=[F|C]/) {
        $<unit> eq 'F' ?? "That's {($<temp> - 32) × .5556}°C"
                       !! "That's { $<temp> × 1.8 + 32   }°F"
    }
}

class BFF { method irc-to-me ($ where /'♥'/) { 'I ♥ YOU!' } }

.run with IRC::Client.new:
    :nick<MahBot>
    :host<irc.freenode.net>
    :channels<#perl6>
    :debug
    :plugins(Trickster, BFF)

<Zoffix> MahBot, time
<MahBot> Zoffix, 2016-07-23T19:59:44.481553-04:00
<Zoffix> MahBot, temp 42F
<MahBot> Zoffix, That's 5.556°C
<Zoffix> MahBot, temp 42C
<MahBot> Zoffix, That's 107.6°F
<Zoffix> MahBot, I ♥ you!
<MahBot> Zoffix, I ♥ YOU!

Outside of the signature, we no longer have any need for the message object, so we use the anonymous $ parameter in its place. We then type-constrain that parameter with a regex match, and so the method will be called only if the text of the message matches that regex. Since no methods will be called on failed matches, we no longer have to mess around with the whole $.NEXT business or compose any roles into our plugins.

The bodies of our methods each have a single statement that produces the response value for the event. In the temperature converter, we use the ternary operator to select which formula to use for the conversion, depending on the unit requested, and yes, the $<unit> and $<temp> captures created in the signature type constraint match are available in the method's body.

An Eventful Day

Along with standard named and numerical IRC protocol events, IRC::Client offers convenience events. One of them we've already seen: the irc-to-me event. Such events are layered, so one IRC event can trigger several IRC::Client's events. For example, if someone addresses our bot in a channel, the following chain of events will be fired:

irc-addressed  ▶  irc-to-me  ▶  irc-privmsg-channel  ▶  irc-privmsg  ▶  irc-all

The events are ordered from "narrowest" to "widest": irc-addressed can be triggered only in-channel, when our bot is addressed; irc-to-me can also be triggered via notice and private message, so it's wider; irc-privmsg-channel includes all channel messages, so it's wider still; and irc-privmsg also includes private messages to our bot. The chain ends by the widest event of them all: irc-all.

If a plugin's event handler returns any value other than $.NEXT, later events in the event chain won't be fired, just as plugins later in the plugin chain won't be tried for the same reason. Each event is tried on all of the plugins, before attempting to handle a wider event.

By setting the :debug attribute to level 3 or higher, you'll get emitted events in the debug output. Here's our bot attempting to handle unknown command blarg and then processing command time handled by irc-to-me event handler we defined:

All of IRC::Client's events have irc- prefix, so you can freely define auxiliary methods in your plugin, without worrying about conflicting with event handlers. Speaking of emitting things...

Keep 'Em Commin'

Responding to commands is sweet and all, but many bots will likely want to generate some output out of their own volition. As an example, let's write a bot that will annoy us whenever we have unread GitHub notifications!

use IRC::Client;
use HTTP::Tinyish;
use JSON::Fast;

class GitHub::Notifications does IRC::Client::Plugin {
    has Str  $.token  = %*ENV<GITHUB_TOKEN>;
    has      $!ua     = HTTP::Tinyish.new;
    constant $API_URL = 'https://api.github.com/notifications';

    method irc-connected ($) {
        start react {
            whenever self!notification.grep(* > 0) -> $num {
                $.irc.send: :where<Zoffix>
                            :text("You have $num unread notifications!")
                            :notice;
            }
        }
    }

    method !notification {
        supply {
            loop {
                my $res = $!ua.get: $API_URL, :headers{ :Authorization("token $!token") };
                $res<success> and emit +grep *.<unread>, |from-json $res<content>;
                sleep $res<headers><X-Poll-Interval> || 60;
            }
        }
    }
}

.run with IRC::Client.new:
    :nick<MahBot>
    :host<irc.freenode.net>
    :channels<#perl6>
    :debug
    :plugins(GitHub::Notifications.new)

[00:25:41] -MahBot- Zoffix, You have 20 unread notifications!
[00:26:41] -MahBot- Zoffix, You have 19 unread notifications!

We create GitHub::Notifications class that does the IRC::Client::Plugin role. That role gives us the $.irc attribute, which is the IRC::Client object we'll use to send messages to us on IRC.

Aside from irc-connected method, the class is just like any other: a public $.token attribute for our GitHub API token, a private $!ua attribute that keeps our HTTP User Agent object around, and a private notification method, where all the action happens.

Inside notification, we create a Supply that will emit the number of unread notifications we have. It does so by using an HTTP::Tinyish object to access a GitHub API endpoint. On line 24, it parses the JSON returned by successful requests, and greps the message list for any items with unread property set to true. The prefix + operator converts the list to an Int that is total items found, which is what we emit from our supply.

The irc-connected event handler gets triggered when we successfully connect to an IRC server. In it, we start an event loop that reacts whenever we receive the current unread messages count from our supply given by notifications method. Since we're only interested in cases where we do have unread messages, we also pop a grep on the supply to filter out the cases without any messages (yes, we could avoid emitting those in the first place, but I'm showing off Perl 6 here 😸). And once we do have unread messages, we simply call IRC::Client's .send method, asking it to send us an IRC notice with the total number of unread messages. Pure awesomeness!

Don't Wait Up

We've covered the cases where we either have an asynchronous supply of values we sent to IRC or where we reply to a command right away. It's not uncommon for a bot command to take some time to execute. In those cases, we don't want the bot to lock up while the command is doing its thing.

Thanks to Perl 6's excellent concurrency primitives, it doesn't have to! If an event handler returns a Promise, the Client Object will use its .result as the reply when it is kept. This means that in order to make our blocking event handler non-blocking, all we have to do is wrap its body in a start { ... } block. What could be simpler?

As an example, let's write a bot that will respond to bash command. The bot will fetch bash.org/?random1, parse out the quotes from the HTML, and keep them in the cache. When the command is triggered, the bot will hand out one of the quotes, repeating the fetching when the cache runs out. In particular, we don't want the bot to block while retrieving and parsing the web page. Here's the full code:

use IRC::Client;
use Mojo::UserAgent:from<Perl5>;

class Bash {
    constant $BASH_URL = 'http://bash.org/?random1';
    constant $cache    = Channel.new;
    has        $!ua    = Mojo::UserAgent.new;

    multi method irc-to-me ($ where /bash/) {
        start $cache.poll or do { self!fetch-quotes; $cache.poll };
    }

    method !fetch-quotes {
        $cache.send: $_
            for $!ua.get($BASH_URL).res.dom.find('.qt').each».all_text.lines.join: '  ';
    }
}

.run with IRC::Client.new:
    :nick<MahBot>
    :host<irc.freenode.net>
    :channels<#perl6>
    :debug
    :plugins(Bash.new)

<Zoffix> MahBot, bash
<MahBot> Zoffix, <Time> that reminds me of when Manning and I installed OS/2 Warp4 on a box and during the install routine it said something to the likes of 'join the hundreds of people on the internet'

For page fetching needs, I chose Perl 5's Mojo::UserAgent, since it has an HTML parser built-in. The :from<Perl5> adverb indicates to the compiler that we want to load a Perl 5, not Perl 6, module.

Since we're multi-threading, we'll use a Channel as a thread-safe queue for our caching purposes. We subscribe to the irc-to-me event where text contains word bash. When the event handler is triggered, we pop out to a new thread using the start keyword. Then we .poll our cache and use the cached value if we have one, otherwise, the logic will move onto the do block that that calls the fetch-quotes private method and when that completes, polls the cache once more, getting a fresh quote. All said and done, a quote will be the result of the Promise we return from the event handler.

The fetch-quotes method fires up our Mojo::UserAgent object that fetches the random quotes page from the website, finds all HTML elements that have class="qt" on them—those are paragraphs with quotes. Then, we use a hyper method call to convert those paragraphs to just text and that final list is fed to our $cache Channel via a for loop. And there you go, we non-blockingly connected our bot to the cesspit of the IRC world. And speaking of things you may want to filter...

Watch Your Mouth!

Our bot would get banned rather quickly if it spewed enormous amounts of output into channels. An obvious solution is to include logic in our plugins that would use a pastebin if the output is too large. However, it's pretty impractical to add such a thing to every plugin we write. Luckily, IRC::Client has support for filters!

For any method that issues a NOTICE or PRIVMSG IRC command, IRC::Client will pass the output through classes given to it via :filters attribute. This means we can set up a filter that will automatically pastebin large output, regardless of what plugin it comes from.

We'll re-use our bash.org quote bot, except this time it will pastebin large quotes to Shadowcat pastebin. Let's look at some code!

use IRC::Client;
use Pastebin::Shadowcat;
use Mojo::UserAgent:from<Perl5>;

class Bash {
    constant $BASH_URL = 'http://bash.org/?random1';
    constant $cache    = Channel.new;
    has        $!ua    = Mojo::UserAgent.new;

    multi method irc-to-me ($ where /bash/) {
        start $cache.poll or do { self!fetch-quotes; $cache.poll };
    }

    method !fetch-quotes {
        $cache.send: $_
            for $!ua.get($BASH_URL).res.dom.find('.qt').each».all_text;
    }
}

.run with IRC::Client.new:
    :nick<MahBot>
    :host<irc.freenode.net>
    :channels<#zofbot>
    :debug
    :plugins(Bash.new)
    :filters(
        -> $text where .lines > 1 || .chars > 300 {
            Pastebin::Shadowcat.new.paste: $text.lines.join: "\n";
        }
    )

<Zoffix> MahBot, bash
<MahBot> Zoffix, <intuit> hmm maybe sumtime next week i will go outside'
<Zoffix> MahBot, bash
<MahBot> Zoffix, http://fpaste.scsys.co.uk/528741

The code that does all the filtering work is small enough that it's easy to miss—it's the last 5 lines in the program above. The :filters attribute takes a list of Callables, and here we're passing a pointy block. In its signature we constraint the text to be more than 1 line or more than 300 characters long, so our filter will be run only when those criteria are met. Inside the block, we simply use the Pastebin::Shadowcat module to throw the output onto the pastebin. Its .paste method returns the URL of the newly-created paste, which is what our filter will replace the original content with. Pretty awesome!

It Spreads Like Butter

In the past, when I used other IRC client tools, whenever someone asked me to place my bots on other servers, the procedure was simple: copy over the code to another directory, change config, and you're done. It almost made sense that a new server would mean a "new" bot: different channels, different nicknames, and so on.

In Perl 6's IRC::Client, I tried to re-imagine things a bit: a server is merely another identifier for a message, along with a channel or nickname. This means connecting your bot to multiple servers is as simple as adding new server configuration via :servers attribute:

use IRC::Client;

class BFF {
    method irc-to-me ($ where /'♥'/) { 'I ♥ YOU!' }
}

.run with IRC::Client.new:
    :debug
    :plugins(BFF)
    :nick<MahBot>
    :channels<#zofbot>
    :servers(
        freenode => %(
            :host<irc.freenode.net>,
        ),
        local => %(
            :nick<P6Bot>,
            :channels<#zofbot #perl6>,
            :host<localhost>,
        )
    )

[on Freenode server]
<ZoffixW> MahBot, I ♥ you
<MahBot> ZoffixW, I ♥ YOU!

[on local server]
<ZoffixW> P6Bot, I ♥ you
<P6Bot> ZoffixW, I ♥ YOU!

First, our plugin remains oblivious that it's being run on multiple servers. Its replies get redirected to the correct server and IRC::Client still executes its method handler in a thread-safe way.

In the IRC::Client's constructor we added :servers attribute that takes a Hash. The keys of this Hash are servers' labels and values are server-specific configurations that override global settings. So freenode server gets its :nick and :channels from the :nick and :channels attributes we give to IRC::Client, while the local server overrides those with its own values.

The debug output now has server lables printed, to indicate to which server the event applies:

And so, but simply telling the bot to connect to another server, we made it multi-server, without making any changes to our plugins. But what do we do when we do want to talk to a specific server?

Send It That Way

When the bot is .run, the Client Object changes the values of :servers attribute to be IRC::Client::Server objects. Those stringify to the label for the server they represent and we can get them either from the .server attribute of the Message Object or .servers hash attribute of the Client Object. Client Object methods such as .send or .join take an optional server attribute that controls which server the message will be sent to and defaults to value *, which means send to every server.

Here's a bot that connects to two servers and joins several channels. Whenever it sees a channel message, it forwards it to all other channels and sends a private message to user Zoffix on server designated by label local.

use IRC::Client;

class Messenger does IRC::Client::Plugin {
    method irc-privmsg-channel ($e) {
        for $.irc.servers.values -> $server {
            for $server.channels -> $channel {
                next if $server eq $e.server and $channel eq $e.channel;

                $.irc.send: :$server, :where($channel), :text(
                    "$e.nick() over at $e.server.host()/$e.channel() says $e.text()"
                );
            }
        }

        $.irc.send: :where<Zoffix>
                    :text('I spread the messages!')
                    :server<local>;
    }
}

.run with IRC::Client.new:
    :debug
    :plugins[Messenger.new]
    :nick<MahBot>
    :channels<#zofbot>
    :servers{
        freenode => %(
            :host<irc.freenode.net>,
        ),
        local => %(
            :nick<P6Bot>,
            :channels<#zofbot #perl6>,
            :host<localhost>,
        )
    }

[on Freenode server/#zofbot]
<ZoffixW> Yey!
[on local server/#zofbot]
<P6Bot> ZoffixW over at irc.freenode.net/#zofbot says Yey!
[on local server/#perl6]
<P6Bot> ZoffixW over at irc.freenode.net/#zofbot says Yey!
[on local server/ZoffixW private message queue]
<P6Bot> I spread the messages!

We subscribe to the irc-privmsg-channel event and when it's triggered, we loop over all the servers. For each server, we loop over all of the connected channels and use $.irc.send method to send a message to that particular channel and server, unless the server and channel are the same as where the message originated.

The message itself calls .nick, .channel, and .server.host methods on the Message Object to identify the sender and origin of the message.

Conclusion

Perl 6 offers powerful concurrency primitives, dispatch methods, and introspection that lets you build awesome non-blocking, event-based interfaces. One of them is IRC::Client that lets you use IRC networks. It's here. It's ready. Use it!

Re-posted from http://rakudo.org/2016/07/22/announce-rakudo-star-release-2016-07/
The release was prepared by Steve 'stmuk' Mynott

On behalf of the Rakudo and Perl 6 development teams, I'm pleased to announce the July 2016 release of "Rakudo Star", a useful and usable production distribution of Perl 6. The tarball for the July 2016 release is available from http://rakudo.org/downloads/star/.

This is the third post-Christmas (production) release of Rakudo Star and implements Perl v6.c. It comes with support for the MoarVM backend (all module tests pass on supported platforms).

Please note that this release of Rakudo Star is not fully functional with the JVM backend from the Rakudo compiler. Please use the MoarVM backend only.

In the Perl 6 world, we make a distinction between the language ("Perl 6") and specific implementations of the language such as "Rakudo Perl". This Star release includes release 2016.07 of the Rakudo Perl 6 compiler, version 2016.07 of MoarVM, plus various modules, documentation, and other resources collected from the Perl 6 community.

Some of the new compiler features since the last Rakudo Star release include:

• Ability to use a custom debugger module
• The "is-approx" sub from Test.pm6 now allows for relative/absolute tolerance
• A fail in a custom BUILD will now be returned, rather than thrown
• Introduce .Map coercer
• Implement alternate ways to call subtest
• Support for new leap-second at the end of 2016
• The "is required" trait on Attributes can now take a Bool or a Str
• IO::[Path,Handle] gained a .mode method which returns the POSIX file permissions
• Distribution is now a role interface that enables encapsulating IO used for distribution installation
• CompUnit::Repository::Installation now uses the new Distribution interface
• Custom repository implementations now supported, including precompilation

Compiler maintenance since the last Rakudo Star release includes:

• Basic object creation (using either .new or .bless) now up to 3x faster
• All routines now have less overhead
• The MMD cache accepts candidates with named parameters if it can. (This made adverbed slices about 18x as fast)
• Sigificant optimizations for speed in many parts of the system (.map, gather/take etc.)
• Many precompilation fixes (including EVAL and improved support of OS packaging)
• Arrays with holes (e.g. from :delete) now correctly iterate/auto-vivify
• An issue with reverse dependencies of installed modules was fixed
• "is_approx" sub (note underscore) from Test.pm6 deprecated
• Harden Mu.Str against moving GC
• Simplify $USER/$GROUP initialization
• Mu can now be the result of a Promise
• samewith() now also works on non-multi's
• Many fixes in the area of pre-compilation and installing modules
• count-only and bool-only now are optional methods in Iterators (only to be implemented if they can work without generating anything)
• IO::ArgFiles.slurp / IO::ArgFiles.eof are fixed
• REPL whitespace and error handling
• CompUnit::Repository::Installation no longer considers bin/xxx and resources/bin/xxx the same content address
• min/max on Failures throw instead of returning ±Inf
• NativeCall's is mangled trait no longer ignored for CPPStruct
• Many Str, List and Array methods much faster
• Map/Hash initializations are now 30% faster
• make DESTDIR now correctly finds CompUnit::Repository::Staging
• Output from Test.pm6's diag() is no longer lost in non-verbose prove output when called at the start of the test file or during TODO tests
• Improved error messages

Notable changes in modules shipped with Rakudo Star:

• DBIish: v0.5.9 (with many Oracle/MySQL fixes) plus README.pod and mojibake fixes
• NativeHelpers-Blob: v0.1.10
• PSGI: v1.2.0 supports P6SGI 0.7Draft
• Pod-To-HTML: v0.1.2 plus fixes
• debugger-ui-commandline: README fixes
• doc: many fixes to documentation content and HTML generation
• panda: Avoid Rakudo internals deprecation warning and don't require Build.pm to inherit Panda::Builder
• perl6-file-which: CI fixes
• perl6-http-easy: v1.1.0 (with more flexible P6SGI support) plus avoid errors in binary request
• shell-command: Mention already implemented commands missing from README
• perl6-lwp-simple: track github.com/perl6/perl6-lwp-simple as upstream (as panda does) which has a test fix needed since we don't support https in R* and a test url had a new https redirect

perl6intro.pdf has also been updated.

There are some key features of Perl 6 that Rakudo Star does not yet handle appropriately, although they will appear in upcoming releases. Some of the not-quite-there features include:

• advanced macros
• non-blocking I/O (in progress)
• some bits of Synopsis 9 and 11
• There is an online resource at http://perl6.org/compilers/features that lists the known implemented and missing features of Rakudo's backends and other Perl 6 implementations.

In many places we've tried to make Rakudo smart enough to inform the programmer that a given feature isn't implemented, but there are many that we've missed. Bug reports about missing and broken features are welcomed at rakudobug@perl.org.

See http://perl6.org/ for links to much more information about Perl 6, including documentation, example code, tutorials, presentations, reference materials, design documents, and other supporting resources. Some Perl 6 tutorials are available under the "docs" directory in the release tarball.

The development team thanks all of the contributors and sponsors for making Rakudo Star possible. If you would like to contribute, see http://rakudo.org/how-to-help, ask on the perl6-compiler@perl.org mailing list, or join us on IRC #perl6 on freenode.

Read this article on Perl6.Party

Be sure to read Part I of this series.

As I was tagging tickets in my bug ticket helper app, I was surprised by how often I was tagging tickets with this particular tag:

It may have been the most used tag of them all. And so, it made be think about...

PART II: The Experienced Contributor

I will be referring to "core developers," but this generally applies to any person who has great familarity with the project, how it should, does, and will work —The Experienced Contributor. When it comes to bug queues, can this type of people do more than just pick the bug they like the most and fix it?

LESSON 3: Many Tickets Can Be Fixed With A Single Comment

On my date with the bug queue, I found many tickets that looked relatively easy to fix, from a technological point of view, but I couldn't even begin working on them for a simple reason: I didn't know what the correct behaviour should be.

Here's a good example:

> "345".Int
345
> "3z5".Int
Cannot convert string to number: trailing characters after number in '3^z5'
> "34\x[308]5".Int
3

When non-numeric characters are found, the conversion from Str to Int fails with an error message, but if a diacritic is present on a digit, the conversion stops and returns only a partial number. The issue sounds simple enough, so let's get cracking? But wait!

What is the correct behaviour here? Should it fail with the same error message as "3z5" fails with, since a diacritic isn't a digit, or should it be silently ignored? After all, the entire string does match \d+:

say "34\x[308]5" ~~ /^ \d+ $/
# OUTPUT:
# ｢34̈5｣

In this case, we have a contributor ready to resolve the ticket, but they are stimied by the lack of direction. When a core member sees a ticket like that, and they don't have the time to fix the bug, they should comment on the ticket, indicating what the expected behaviour is. Now, a different contributor who does have the time can proceed to fix the bug with great confidence their patch will be accepted.

And just like that, with a single comment from a core developer the ticket became much easier to fix! And speaking of easy things to fix...

LESSON 4: Tag Your Tickets and Make Them Easy to Find!

When it comes to your bug ticket queue, this is probably something you don't want to see:

<BrokenRobot> [Coke]: I've no idea how to locate them and now RT pissed me off enough that I gone off the idea of doing those tickets ~_~
...
<BrokenRobot> 18 minutes to find tickets. Retarded

And when a contributor is trying to find some bugs to fix, this is something they don't want to see:

This isn't the interface for the latest NASA space probe. It's the search page for the RT ticket system. And that's not even the full story, because if the contributor wants to find tagged tickets, they have to use "Advanced" version that is nothing more than a textbox asking for what may or may not be a SQL query.

Sure, if you've used it a few times, it becomes simple enough to use, but a new contributor who's beggining to learn the guts of your project probably doesn't want to spend their entire afternoon learning how to search for tickets. They probably wouldn't be too sure about what to search for anyway.

So what's the solution? Tags! Categorize your tickets with tags and then you can have a Wiki or page on your project's website linking to search queries for specific tags. Here are some ideas to consider when deciding on ticket tags:

For Core Member Review / Bikeshed

Consider this comment on a ticket:

This should probably return a List instead of a Hash.

The problem with it is that it's unclear whether or not there needs to be a discussion or if the ticket should be resolved following the suggestion. So you have one group of people who wait for more comments before doing anything and another group who agree with the suggestion and move on without further comments. In the end, you have a stalled ticket and nothing gets fixed.

For that reason, you should use more direct language in your comments. State explicitly whether you're just sharing your opinion/vote on the issue, inviting discussion, or giving direction for how the ticket should be resolved.

For core member reviews, discussions, and general bikeshed, tags can be very useful. For example, the Mojolicious project uses needs feedback Issue labels to invite discussion from its users. The Perl 6 ticket queue has [@LARRY] tag for tickets that need review by the core team members—and core members should review such tickets regularly, because as we've learned in the previous Lesson, some bugs are just a comment away from being fixed!

Area of Expertise

Different issues may require different expertise. For example, an issue with the Perl 6 documentation website may require someone experienced with:

• Design: the look of the website
• UX: usability of the website
• CSS/HTML/JS: the front-end of the website
• Perl 6: scripts for site generation
• Advanced Perl 6: documenting arcane features
• Pod6: documentation format writing and parsing
• English: ensuring correct grammar and good writing style
• Other languages: translation

All that just for a static website. Imagine one that's driven by a web app with a SQL backend!

Not everyone will be an expert in all the areas, so tagging by area of expertise will let a person pick tickets that are easy for them to resolve, depending on what sort of knowledge and training they possess.

Easy / Low Hanging Fruit

These are pretty common in projects: tags that indicate the ticket is very easy to solve. This is typically what new contributors will look for.

However, developers often mistake this tag to mean so easy a blind monkey could do it, and the result potential new contributors see when they view the tag is often:

Easy tickets should not mean brain-dead tickets. Text editing is a task I often see listed as Low Hanging Fruit tickets, but it's booooring. New contributors don't want to get bored. They want to fix things. And there's nothing wrong with requiring to do a bit of learning while they're at it.

The tickets tagged as easy can simply have a comment narrowing down the potential hiding spot of the bug with, perhaps, some guidance on how to debug and fix it. And speaking of guidance...

LESSON 5: Mentoring Converts New Contributors into Experienced Contributors

So you have 20 devs on the team and 1,000 tickets in the bug queue. With 50 tickets per a pair of hands, you sure are busy! The devs never see the light of day and rarely appear on chat channels or comment on mailing lists, bug tickets, and other commentables. That's what busy people do.

However, what would happen if you also spend a bit of time writing tutorials and project internals courses, or training specific people?

<psch> i can also guide you how i figured out what to change where to throw a typed Exception there, because as i said, i think it's a great ticket for getting used to the Grammar and Actions
<BrokenRobot> psch: sure, I'd love to learn that
<psch> BrokenRobot: the most important thing is --target=parse
BrokenRobot: as in, the start for any bug that most likely is Grammar or
Actions based is by isolating it as far as possible - in this case a bare
'&0' - and running that through ./perl6 --target=parse -e
https://gist.github.com/peschwa/1e6a9f84a4c9e67638ff93e5b79f86d9 # like this
EXPR is a scary place, i don't go there
so it's about variable
<psch> https://github.com/rakudo/rakudo/blob/nom/src/Perl6/Grammar.nqp#L2017 so this
<BrokenRobot> So needs to exclude '&' on here?
https://github.com/rakudo/rakudo/bl​ob/nom/src/Perl6/Grammar.nqp#L2025

It's the old adage: teach a person to fish... and you'll turn a profit. Don't think of it as a team of 20 devs. Think of it as a team of 20 devs and several trainees. By dedicating some time on training new contributors you'll be growing your team and reducing the number of bugs per pair of hands. The time you won't spend on fixing bugs right now is an investment into people who will fix more bugs for you later on.

Conclusion

And so ends our date with the bug queue. Whether you're a newbie contributor or a seasoned core hacker, the bug queue is a valuable place to spend your time at.

New contributors help core developers by filtering out the queue, doing preliminary debugging, and writing tests. Core developers help new contribitors by tagging tickets, giving direction for how tickets should be resolved, and providing training. This in turn, lets new contributors help back by fixing bugs and... eventually becoming core contributors themselves.

Read this article on Perl6.Party

Recently, I decided to undertake a quick little journey down to the Perl 6's Bug Queue. A quest for fame and profit—some easy game to hunt for sport. There's plenty of tickets, so how hard can it be? The quick little journey turned out to be long and large, but I've learned a lot of good lessons in the process.

PART I: The Newbie Contributor

Right away, I hit a snag. Some tickets looked hard. On some, it wasn't clear what the correct goal was. And some looked easy, but I wasn't sure whether I wanted to work on them just yet. While the ticket queue has the tag system, I needed some personal tags. Something special just for me....

The Ticket Trakr

So I wrote a nice little helper app—Ticket Trakr. It fetches all the tickets from the bug queue onto one page and lets me tag each of them with any combination of:

• Try to fix
• Easy
• Tests needed
• Needs core member decision
• Needs spec decision
• Check ticket later
• Needs checking if it's still broken
• Too hard
• Not interested

The app worked great! I quickly started going through the queue, looking over the tickets, testing if the bugs were still there, and estimating whether I could and wanted to fix them. And after a full weekend of clicking, tagging, testing, closing, taking an occasional break to hunt bears with a mortar, more closing, testing, tagging, and clicking, I... was through just 200 tickets, which is only 15% of the queue:

And so I've learned the first lesson.

LESSON 1: Going Through Bug Reports is a Full Time Job

Whenever I see someone ask how they can contribute, the basket of offers they receive generally contains: docs, marketing, modules and libraries, or bug fixing. Going through the ticket queue doesn't seem to be considered a task on itself. The ticket queue is just a place where you find the bugs to fix, right?

What may not be obvious is the bug queue contains an extraordinary amount of work that can be performed by less skilled contributors to make it easier for highly-skilled—and by extension much scarcer—contributors to fix bugs. Let's see what those are:

Deciding On Whether The Report Should Be Accepted

Just because you have 1,000 tickets in your bug queue doesn't mean you have 1,000 bugs. Here are some things that might end up as a ticket and how you can help:

• Poor bug reports: you should reply, asking for a decent test case or the missing information
• Bug reports for an unrelated project: move them (or, for the lazy, just close with explanation)
• Feature proposals: ping core members and users for discussion
• A feature confused for a bug: explain the confusion; add to the documentation if this confusion happens often
• Incorrectly used code that was never meant to work: offer a correct example; improve documentation, if needed
• People asking for help with their code: redirect to appropriate help channels; improve documentation, if this happens often
• Patches for other bugs: apply the patches, move them to the appropriate ticket, or make them easier to merge (e.g. make a Pull Request)
• Duplicate bug reports: point out the dupe and close the report
• Spam: grab some white bread and have a lunch break

This is a lot of work, but this is just the basics. What else can a person new to the project can contribute?

Debugging

So we've cleaned up our queue and now we have a few reports that appear to have at least some sort of a six-legged quality to them. Sure, we're new to the project and don't even know where to begin fixing them, but that doesn't mean we can't play around with code to narrow down the problem.

Reproduce the bug

Merely being able to reproduce the bug shows it's likely is indeed a bug and not just a quirk of the reporter's system. Speaking of systems: test the bug on several, especially if you have access to esoteric ones.

Test different versions of the project to see where the bug appeared. If possible, try to narrow down the commit that introduced the bug. For Rakudo Perl 6 compiler, you can use bisectable bot on IRC:

<Zoffix> bisect: say ^10 .grep: { last if * > 2 }
<bisectable> Zoffix: exit code is 0 on both starting points, bisecting by using the output
<bisectable> Zoffix: (2016-03-18) https://github.com/rakudo/rakudo/commit/6d120ca

Even if you can't fix the bug, all this extra information can help a more knowledgeable contributor. Especially if they are the author of the commit that introduced the bug.

Reduce the amount of code reproducing the bug

I've seen a disturbing amount of people playing code golf. Here is the perfect place to put those skills to good use: reproduce the bug with less code. This narrows down the areas where the bug is hiding.

For example, here's the original [actual] reported code, along with the bug report title:

# You can't catch Proc::Async failing because the external program
# doesn't exist if you open it for writing:

perl6 -e 'my $p; try {$p = Proc::Async.new(:w, "asdfcat"); CATCH {die "in
    new"}}; my $pr; try {$pr = $p.start; CATCH {die "in start"}}; try
    {await($p.write("hi\n".encode)); CATCH {die "in write"}}; try
    {$p.close-stdin; CATCH {die "in close"}}; try {await($pr); CATCH
    {die "in await"}}'

That code is a bitch to read, let's pop open an editor and format it properly:

my ( $p, $pr );
try { CATCH { die "in new"   }; $p  = Proc::Async.new: :w, "asdfcat" }
try { CATCH { die "in start" }; $pr = $p.start                       }
try { CATCH { die "in write" }; await $p.write: "hi\n".encode        }
try { CATCH { die "in close" }; $p.close-stdin                       }
try { CATCH { die "in await" }; await $pr                            }

# Outputs nothing when run

That's much better! So the report claims we can't catch things and we've got five try blocks and no output. Hmmm... Let's get rid of all the tries and catching and see what error the write throws:

given Proc::Async.new: :w, "asdfcat" {
    .start;
    await .write: "hi\n".encode;
}

# Outputs nothing when run

And the error is... nothing?! There's no output from the program, so maybe it "succeeds" in writing things and there's nothing to throw? Let's toss in a couple of say calls:

given Proc::Async.new: :w, "asdfcat" {
    say 'Starting';
    .start;

    say 'Writing';
    await .write: "hi\n".encode;

    say 'Wrote';
}

# OUTPUT:
# Starting
# Writing

The Wrote is missing from the output. The original report is incorrect in its assumptions that the issue is with the CATCH block! There's nothing to catch in the first place and the .write Promise seems to exit the program.

Perhaps, CATCH blocks were scary to you, but fixing a bug in a .write call is less so. And there you go: we found a contributor to fix the bug!

Write Tests For The Bug

So you've found out that it is a bug for sure and you've played around with code and maybe even found the commit that broke it. You don't know how to fix the bug, so are we done with this ticket then? How about we write tests!

After the bug is fixed, the project's test suite should contain a test checking regression of that bug. Perl's TAP supports TODO and SKIP features. TODO expects the test to fail and will alert you when it starts to pass. SKIP marks the needed number of tests as skipped and your test logic can avoid running them. So even if the bug is not yet fixed, we can still add tests for it—we'll just TODO or SKIP them, depending on how severe the bug is.

Perl 6's test suite has a fudging mechanism that lets you mark tests as skip or todo, depending on the compiler and virtual machine being tested:

#?rakudo.moar 2 todo 'Waiting fix for RT128526'
ok $fixed,      'stuff is fixed';
ok $also-fixed, 'other stuff is also fixed';

The test suite will expect the above two tests to fail for Rakudo compiler running on the MoarVM virtual machine and it will complain when they start to pass. If someone fixed the bug and wasn't aware of the open ticket, running the test suite alerts them of the ticket automatically. Magic!

As a bonus, the tests are the best description of the bug and they also can expose alternate failure modes that aren't apparent from the bug report itself:

<lizmat> .tell Zoffix the tests you added to S29-os/system.t yesterday hang on OSX  :-(

There's lots to do in the bug queue, but it's not as dull and boring at it appears at the first sight. And so, the bug queue taught me another lesson...

LESSON 2: The Bug Queue Is Not Thankless Labour

If you're a new contributor and you want to get up to speed, the bug queue isn't a bad place to start your learning. And there's a lot to learn from it!

Real-World Usage

You get to see real world code that uses the project you want to learn. Not the out-of-context code in the documentation. Not the polished code in the pre-made examples, but real-life, less-than-perfect, hacked-from-the-hip code. You also get to see code from multiple, vastly different people who use different style guidelines. Being able to easily read code regardless of the style used is a valuable skill.

Esoteric Features and Constructs

Everyone and their brother knows about and and what it's for, but did you know about andthen? I get flashbacks whenever I see it, but it's a useful chaining operator in Perl 6! And I learned about it from a bug report.

You can get very far with language primitives, but knowing more powerful and more appropriate contructs and features will let you write more concise and eloquent code. The bug queue can teach them to you.

Learning The Language

Writing tests can teach you the language you're writing them in. Writing tests for the Perl 6 compiler can teach you Perl 6. How much will you need to think when writing a program that runs a piece of code that might hang and in a set amount of time tells you whether the code resulted in success and kills the thread if it hung? I can do it with my eyes closed now.

Writing tests is a skill in itself and the bug queue gives you lots of opportunity for practice.

Getting To Know People

Going through the bug queue has a large social aspect to it as well:

• Communicating with core members to find out whether and how a ticket should be resolved
• Communicating with ticket creators (members of the community)
• Having (hopefully amicable) discussions about features
• Steering an overly-heated discussion to a more peaceful direction—bugs are annoying, and it's not uncommon for tickets to be written by pissed off people

Also, going through the ticket queue is not the favourite of the people, and the person who does it isn't exactly hated.

Conclusion

The bug queue is a much more complex beast than people may suspect. It has a lot to offer and a lot to teach. From language use, to the quirks of squishy humans. Today, we've examined the vast potential for contribution the ticket queue offers to people new to the project. Helping sift out actual bugs from false reports. Helping with debugging. Helping with tests.

That's not the only thing the bug queue has on the menu. In the next post, we'll examine what it has to teach more experienced regulars and core members of the project! How can you get more bugs fixed? Maybe the bug queue knows the answer.