Results matching “smartmatch”

Perl 5 Porters Mailing List Summary: November 13th-20th

By Sawyer X on November 21, 2017 5:24 PM under p5p-summary

Hey everyone,

Following is the p5p (Perl 5 Porters) mailing list summary for the past week.

Enjoy!

0 comments

COMPLETION Report / Perl 6 IO TPF Grant

By Zoffix Znet on May 29, 2017 11:29 PM under Perl 6, Perl 6, perl6

This document is the May, 2017 progress report for TPF Standardization, Test Coverage, and Documentation of Perl 6 I/O Routines grant. I believe I reasonably satisfied the goals of the grant and consider it completed. This is the final report and may reference some of the work/commits previously mentioned in monthly reports.

Thank You!

I'd like to thank all the donors that support The Perl Foundation who made this grant possible. It was a wonderful learning experience for me, and it brings me joy to look back and see Perl 6 improved due to this grant.

Thank You!

Completeness Criteria

Here are the original completeness criteria (in bold) that are listed on the original grant proposal and my comments on their status:

rakudo repository will contain the IO Action Plan document and it will be fully implemented. The promised document exists. It's fully implemented except for three items that I listed on the IO Action Plan, but which are currently a bit beyond my skill level to implement. I hope to do them eventually, but outside the scope of this grant. They are:
- IO::Handle's Closed status. My original proposal would cause some perfomance issues, so it was decided to improve MoarVM errors instead.
- Optimize multiple stat calls. This involves creating a new nqp op, with code for it implemented in MoarVM and JVM backends.
- Use typed exceptions instead of X::AdHoc. I made typed exceptions be thrown whereever I could. The rest require VM-level exceptions and is on the same level as the handle closed status issue (first item above).
All of the I/O routines will have tests in roast and documented on docs.perl6.org. If any of the currently implemented but unspecced routines are decided against being included in Perl 6 Language, their implementation will no longer be available in Rakudo. To the best of my knowledge, this is completed in full.
The test coverage tool will report all I/O routines as covered and the information will be visible on perl6.wtf (Perl 6's Wonderful Test Files) website. Note: due to current experimental status of the coverage tool, its report may still show some lines or conditionals untested despite them actually being tested; however, it will show the lines where routines' names are specified as covered. To the best of my knowledge, all IO routines currently have tests covering them. Due to its experimental status, the coverage tool shows some attributes as uncovered. I did manually verify all the attributes/routines whose names the tool shows as uncovered contain tests for them. One exception is IO::Notification type (and IO::Path.watch method). While it has full coverage for OSX operating system, it lacks it for other OSes. I tried writing some tests for it, but it looks like the behaviour of the nqp op handling these is broken on Linux and the class needs more work.

Extra Deliverables

I produced these extra deliverables while working on the grant:

The Definitive I/O Guide. Providing tutorial-like documentation for Perl 6's I/O, including documenting some of the bad practices I noticed in the ecosystem (and even a Perl 6 book!) and the correct way to perform those tasks. (N.B. as I write this report, the guide could still use a few extra sections to be considered "The Definitive"; I'll write them in upcoming weeks)
Performance improvements. I made 23 performance-enhancing commits, with many commits making things more than 200% faster, with highest improvement making a routine 6300% faster.
Trait::IO module. Provides does auto-close pseudo-trait to simplify closing of IO handles.
IO::Path::ChildSecure module. Due to large ecosystem usage, IO::Path.child was left as is until 6.d language, at which point it will be made secure (as outlined in the IO Plan). This module provides the secure version in the mean time.
IO::Dir module. Provides IO::Path.dir-like functionality, with ability to close open directory without needing to fully exhaust the returned Seq.
Die module. Implements Perl-5-like behaviour for &die routine.
The "Map of Perl 6 Routines" (or rather the "table") is available on map.perl6.party with its code in perl6/routine-map repo. In near future, I plan to use it to identify incorrect or incomplete entries in our documentation

In addition, I plan to complete these modules some time in the future; the ideas for them were birthed while working on the grant: - NL module. Targeted for use in one liners, the module will provide $*NL dynvar that behaves like Perl 5's $. variable (providing current $*ARGFILES's file's line number). Its implementation became possible thanks to newly-implemented IO::CatHandle type - FastIO module. A re-imagination of core IO, the biggest part of which will be the removal of (user-exposed) use of IO::Spec::* types and $*SPEC variable, which—it is believed—will provide improved performance over core IO. The module is a prototype for some of the proposals that were made during the IO grant and if it offers significant improvements over core IO, its ideas will be used by core IO in future language versions.

Work Performed in May

For the work done in May, many of my commits went into going through the IO routine list, and adding missing tests and documentation, along with fixing bugs (and reporting new ones I found).

The major work was implementation of the IO::CatHandle class that fixed all of the bugs and NYIs with the $*ARGFILES. This work saw the addition of 372 lines of code, 800 lines of tests and 793 lines of documentation.

Work by Other Core Members

jnthn++ completed the handle encoding refactor that will eventually let us get rid of using libuv for syncronous IO and, more importantly, allow us to support user-defined encoders/decoders.

Along with fixing a bunch of bugs, this work altered the performance landscape for IO operations (i.e. some operations may now be a bit faster, others a bit slower), though overall the performance appeared to stay the same.

Tickets Fixed

RT#130456 $*HOME blows up if HOME isn't set
RT#126935 bad .perl for paths with pipe characters (was already fixed; just added an extra test)
RT#131185 IO::Path.perl doesn't roundtrip
RT#130454 tmpdir tries to change the current working directory (routine was removed as part of IO work)
RT#130455 Should I be able to change the temporary directory?
RT#130715 IO::Handle::close shouldn't decide what's a failure (not a bug, but user's confusion; explained and rejected)
RT#131242 IO::Path.move/.copy hangs/trashes file when target/source are same

Grant Commits

During this grant, I've made 417 commits, that are: 134 Rakudo commits + 23 performance-enchancing Rakudo commits + 114 Perl 6 Specification commits + 146 documentation commits,

Performance Rakudo Commits

I've made 23 performance enchancing commits to Rakudo's repository:

4032953 Make IO::Handle.open 75% faster
dcf1bb2 Make IO::Spec::Unix.rel2abs 35% faster
c13480c IO::Path.slurp: make 12%-35% faster; propagate Failures
0e36bb2 Make IO::Spec::Win32!canon-cat 2.3x faster
c6fd736 Make IO::Spec::Win32.is-absolute about 63x faster
894ba82 Make IO::Spec::Win32.split about 82% faster
277b6e5 Make IO::Spec::Unix.rel2abs 2.9x faster
74680d4 Make IO::Path.is-absolute about 80% faster
ff23416 Make IO::Path.is-relative about 2.1x faster
d272667 Make IO::Spec::Unix.join about 40% faster
50429b1 Make IO::Handle.put($x) about 5%-35% faster
204ea59 Make &say(**@args) 70%− faster
6d7fc8e Make &put(**@args) up to 70% faster
76af536 Make 1-arg IO::Handle.say up to 2x faster
aa72bde Remove dir's :absolute and :Str; make up to 23% faster
48cf0e6 Make IO::Spec::Cygwin.is-absolute 21x faster
c96727a Fix combiners on SPEC::Win32.rel2abs; make 6% faster
0547979 Make IO::Spec::Unix.path consistent and 4.6x faster
8992af1 Fix IO::Spec::Win32.path and make 26x faster
7d6fa73 Make IO::Spec::Win32.catpath 47x faster
494659a Make IO::Spec::Win32.join 26x faster
6ca702f Make IO::Spec::Unix.splitdir 7.7x faster
2816ef7 Make IO::Spec::Win32.splitdir 25x faster

Non-Performance Rakudo Commits

Other than perf commits, I've also made 134 commits to the Rakudo's repository:

dd4dfb1 Fix crash in IO::Special .WHICH/.Str
76f7187 Do not cache IO::Path.e results
212cc8a Remove IO::Path.Bridge
a01d679 Remove IO::Path.pipe
55abc6d Improve IO::Path.child perf on *nix
4fdebc9 Make IO::Spec::Unix.split 36x Faster
0111f10 Make IO::Spec::Unix.catdir 3.9x Faster
fa9aa47 Make R::I::SET_LINE_ENDING_ON_HANDLE 4.1x Faster
c360ac2 Fix smartmatch of Cool ~~ IO::Path
0c7e4a0 Do not capture args in .IO method
9d8d7b2 Log all changes to plan made during review period
87987c2 Removerole IOand its .umask method
36ad92a Remove 15 methods from IO::Handle
a5800a1 Implement IO::Handle.spurt
aa62cd5 Remove &tmpdir and &homedir
a0ef2ed Improve &chdir, &indir, and IO::Path.chdir
ca1acb7 Fix race in &indir(IO::Path …)
2483d68 Fix regression in &chdir's failure mode
5464b82 Improve &*chdir
4c31903 Add S32-io/chdir-process.t to list of test files to run
cb27bce Clean up &open and IO::Path.open
099512b Clean up and improve all spurt routines
b62d1a7 Give $*TMPDIR a container
b1e7a01 Implement IO::Path.extension 2.0
15a25da Fix ambiguity in empty extension vs no extension
50aea2b Restore IO::Handle.IO
966a7e3 Implement IO::Path.concat-with
94a6909 Clean up IO::Spec::Unix.abs2rel a bit
a432b3d Remove IO::Path.abspath (part 2)
954e69e Fix return value of IO::Special methods
67f06b2 Run S32-io/io-special.t test file
a0b82ed Make IO::Path::* actually instantiate a subclass
0c8bef5 Implement :parent in IO::Spec::Cygwin.canonpath
0a442ce Remove type constraint in IO::Spec::Cygwin.canonpath
b4358af Delete code for IO::Spec::Win32.catfile
e681498 Make IO::Path throw when path contains NUL byte
6a8d63d Implement :completely param in IO::Path.resolve
b6838ee Remove .f check in .z
184d499 Make IO::Handle.Supply respect handle's mode
f1b4af7 Implement IO::Handle.slurp
90da80f Rework read methods in IO::Path/IO::Handle
8c09c84 Fix symlink and link routines
da1dea2 Fix &symlink and &link
7f73f92 Make IO::Path.new-from-absolute-path private
ff97083 Straighten up rename, move, and copy
0d9ecae Remove multi-dir &mkdir
6ee71c2 Coerce mode in IO::Path.mkdir to Int
d46e8df Add IO::Pipe .path and .IO methods
c01ebea Make IO::Path.mkdir return invocant on success
1f689a9 Fix up IO::Handle.Str
490ffd1 Do not use self.Str in IO::Path errors
40217ed Swap .child to .concat-with in all the guts
fd503f8 Revert "Removerole IOand its .umask method"
c95c4a7 Make IO::Path/IO::Special do IO role
214198b Implement proper args for IO::Handle.lock
9a2446c Move Bool return value to signature
51e4629 Amend rules for last part in IO::Path.resolve
b8458d3 Rewordmethod childfor cleaner code
1887114 Implement IO::Path.child-secure
9d8e391 Fix IO::Path.resolve with combiners; timotimo++
0b5a41b Rename IO::Path.concat-with to .add
a98b285 Remove IO::Path.child-secure
8bacad8 Implement IO::Path.sibling
7112a08 Add :D on invocant for file tests
b2a64a1 Fix $*CWD inside IO::Path.dir's :test Callable
6fa4bbc Straighten out &slurp/&spurt/&get/&getc/&close
34b58d1 Straighten out &lines/&words
d0cd137 Make dir take any IO(), not just Cool
7412184 Make $*HOME default to Nil, not Any
475d9bc Fix display of backslashes in IO::Path.gist
6ef2abd Revert "Fix display of backslashes in IO::Path.gist"
134efd8 Fix .perl for IO::Path and subclasses
69320e7 Fix .IO on :U of IO::Path subclasses
eb8d006 Make IO::Handle.iterator a private lines iterator
08a8075 Fix IO::Path.copy/move when source/target are same
973338a Fix IO::Handle.comb/.split; make them .slurp
b43ed18 Make IO::Handle.flush fail with typed exceptions
276d4a7 Remove .tell info in IO::Handle.gist
f4309de Fix IO::Spec::Unix.is-absolute for combiners on /
06d8800 Fix crash when setting .nl-in ...
7e9496d Make IO::Handle.encoding settable via .new
95e49dc Make IO::Handle.open respect attribute values
6ed14ef Remove:directoryfrom IO::Spec::*.split
9021a48 Make IO::Path.parts a Map instead of Hash
a282b8c Fix IO::Handle.perl.EVAL roundtrippage
a412788 Make IO::Path.resolve set CWD to $!SPEC.dir-sep
84502dc Implement $limit arg for IO::Handle.words
613bdcf Make IO::Handle.print/.put sig consistent
0646d3f Allow no-arg &prompt
4a8aa27 Implement IO::CatHandle.close
4ad8b17 Implement IO::CatHandle.get
3b668b6 Implement IO::CatHandle.getc
25b664a Implement IO::CatHandle.words
7ebc386 Implement IO::CatHandle.slurp
52b34b7 Implement IO::CatHandle.comb/.split
beaa925 Implement IO::CatHandle.read
ccc90fd Implement IO::CatHandle.readchars
40f4dc9 Implement IO::CatHandle.Supply
0c9aea7 Implement IO::CatHandle.encoding
ee1e185 Implement IO::CatHandle.eof
80686a7 Implement IO::CatHandle.t/.path/.IO/.native-descriptor
993de50 Implement IO::CatHandle.gist/.Str/.opened/.open
677c4ea Implement IO::CatHandle.lock/.unlock/.seek/.tell
e657ed1 Implement IO::CatHandle.chomp/.nl-in
a452e42 Implement IO::CatHandle.on-switch
f539a62 Swap IO::ArgFiles to IO::CatHandle impl
fa7aa1c Implement IO::CatHandle.perl method
21fd2c4 Remove IO::Path.watch
65941b2 Revert "Remove IO::Path.watch"
a47a78f Remove useless :SPEC/:CWD on some IO subs
d13d9c2 Throw out IO::Path.int

Perl 6 Specification Commits

I've made 114 commits to the Perl 6 Specification (roast) repository:

63370fe Test IO::Special .WHICH/.Str do not crash
465795c Test IO::Path.lines(*) does not crash
091931a Expand &open tests
8d6ca7a Cover IO::Path.ACCEPTS
14b6844 Use Numeric instead of IO role in dispatch test
5a7a365 Expand IO::Spec::*.tmpdir tests
f48198f Test &indir
bd46836 Amend &indir race tests
04333b3 Test &indir fails with non-existent paths by default
73a5448 Remove two fudged &chdir tests
86f79ce Expand &chdir tests
430ab89 Test &*chdir
86c5f9c Delete qp{} tests
3c4e81b Test IO::Path.Str works as advertised
ba3e7be Merge S32-io/path.t and S32-io/io-path.t
79ff022 Expand &spurt and IO::Path.spurt tests
1d4e881 Test $*TMPDIR can betemped
b23e53e Test IO::Path.extension
2f09f18 Fix incorrect test
305f206 Test empty-string extensions in IO::Path.extension
0e47f25 Test IO::Path.concat-with
e5dc376 Expand IO::Path.accessed tests
43ec543 Cover methods of IO::Special
bd8d167 Test IO::Path::* instantiate a subclass
d8707e7 Cover IO::Spec::Unix.basename
c3c51ed Cover IO::Spec::Win32.basename
896033a Cover IO::Spec::QNX.canonpath
7c7fbb4 Cover :parent arg in IO::Spec::Cygwin.canonpath
8f73ad8 Change \0 roundtrip test to \t roundtrip test
b16fbd3 Add tests to check nul byte is rejected
ee7f05b Move is-path sub to top so it can be reused
a809f0f Expand IO::Path.resolve tests
feecaf0 Expand file tests
a4c53b0 Use bin IO::Handle to test its .Supply
7e4a2ae Swap .slurp-rest to .slurp
d4353b6 Rewrite .l on broken symlinks test
416b746 Test symlink routines
8fa49e1 Testlinkroutines
637500d Spec IO::Pipe.path/.IO returns IO::Path type object
64ff572 Cover IO::Path/IO::Pipe's .Str/.path/.IO
4194755 Test IO::Handle.lock/.unlock
a716962 Amend rules for last part in IO::Path.resolve
f3c5dae Test IO::Path.child-secure
92217f7 Test IO::Path.child-secure with combiners
39677c4 IO::Path.concat-with got renamed to .add
7a063b5 Fudge .child-secure tests
3b36d4d Test IO::Path.sibling
41b7f9f Test $*CWD in IO::Path.dir(:test) Callable
18d9c04 Cover IO::Handle.spurt
8f78ca6 Test &words with IO::ArgFiles
ea137f6 Cover IO::Handle.tell
71a6423 Add $*HOME tests
95d68a2 Test IO::Path.gist does escapes of backslashes
de89d25 Revert "Test IO::Path.gist does escapes of backslashes"
9e8b154 Test IO::Handle.close can be...
853f76f Test IO::Pipe.close returns pipe's Proc
d543e75 Test IO::Handle.DESTROY closes the handle
1ed18b4 Add test for .perl.EVAL roundtrip with combiners
704210c Test we can roundtrip IO::Path.perl
2689eb1 Test .IO on :U of IO::Path subclasses
40353f1 Test for IO::Handle:D { ... } loops over handle
4fdb850 Test IO::Path.copy/move when source/target are same
98917dc Test IO::Path.dir's absoluteness behaviour
71eebc7 Test IO::Spec::Unix.extension
4495615 Test IO::Handle.flush
60f5a6d Test IO::Handle.t when handle is a TTY
31e3993 Test IO::Path*.gist
c481433 Test .is-absolute method for / with combiners
8ee0a0a Test IO::Spec::Win32.rel2abs with combiners
a41027f Test IO::Handle.nl-in can be set
e82b798 Test IO::Handle.open respects attributes
2c29150 Test IO::Handle.nl-in attribute
03ce93b Test IO::Handle.encoding can be set
8ae81c0 Test no-arg candidate of &note
fb61306 Test IO::Path.parts attribute
7266522 Test return type of IO::Spec::Unix.path
6ac3b4a Test IO::Spec::Win32.path
dbbea15 Test IO::Handle.perl.EVAL roundtrips
5eb513c Test IO::Path.resolve sets CWD to $!SPEC.dir-sep
b0c4a7a Test &words, IO::Handle.words, and IO::Path.words
f3d1f67 Test $limit arg with &lines/IO::*.lines
4f5589b Add test for handle leak in IO::Path.lines
4d0f97a Add &put/IO::Handle.put tests
125fe18 Add &prompt tests
939ca8d Test IO::CatHandle.close
9833012 Test IO::CatHandle.get
2f65a72 Test IO::CatHandle.getc
a4a7eaa Test IO::CatHandle.words
1131c09 Add &put/IO::Handle.put tests
80de9b6 Add &prompt tests
bacfd9f Test IO::CatHandle.slurp
e78e3c0 Test IO::CatHandle.comb/.split
f1c1125 Test IO::CatHandle.read
e9e78e1 Test IO::CatHandle.readchars
0479087 Test IO::CatHandle.Supply
71953e3 Test IO::CatHandle.encoding
db4847e Test IO::CatHandle.eof
175ba45 Test IO::CatHandle.t/.path/.IO/.native-descriptor
c6cc66a Test IO::CatHandle.gist/.Str/.opened/.open
dcdac1a Test IO::CatHandle.lock/.unlock/.seek/.tell
f48c26e Test IO::CatHandle.chomp/.nl-in
8afd758 Test IO::CatHandle.DESTROY
c7eff2b Test IO::CatHandle.on-switch
e87e20d Test IO::CatHandle.next-handle
28717f0 Test IO::CatHandle.perl method
432bf94 Test IO::Path.watch
ce1b637 Test IO::Handle.say
0bb6298 Test IO::Handle.print-nl
47c88ab Test IO::Pipe.proc attribute
945621d Test IO::Path.SPEC attribute
5fb4b63 Test IO::Path.CWD/.path attributes
d0e5701 Test IO::Path.Numeric and other .numeric methods
94d7133 Test 0-arg &say/&put/&print
38c61cd Test &slurp() and &slurp(IO::Handle)

Perl 6 Documentation Commits

I've made 146 commits to the Perl 6 Documentation repository:

fd7a41b Improve code example
110efb4 No need for.ends-with``
69d32da Remove IO::Handle.z
d02ae7d Remove IO::Handle.rw and .rwx
ccae74a Fix incorrect information for IO::Path.absolute
3cf943d Expand IO::Path.relative
cc496eb Remove mention of IO.umask
335a98d Remove mention ofrole IO``
cc6539b Remove 8 methods from IO::Handle
0511e07 Document IO::Spec::*.tmpdir
db36655 Remove tip to use $*SPEC to detect OS
839a6b3 Expand docs for $*HOME and $*TMPDIR
d050d4b Remove IO::Path.chdir prose
1d0e433 Document &chdir
3fdc6dc Document &*chdir
e1a299c Reword "defined as" for &*chdir
e5225be Fix URL to &*chdir
bf377c7 Document &indir
5aa614f Improve suggestion for Perl 5's opendir
a53015a Clarify value of IO::Path.path
bdd18f1 Fix desc of IO::Path.Str
b78d4fd Include type names in links to methods
b8fba97 Point out my $*CWD = chdir … is an error
d5abceb Write docs for all spurt routines
b9e692e Document new IO::Path.extension
65cc372 Document IO::Path.concat-with
24a6ea9 Toss all of the TODO methods in IO::Spec*
1f75ddc Document IO::Spec*.abs2rel
cc62dd2 Kill IO::Path.abspath
1973010 Document IO::Path.ACCEPTS
b3a9324 Expand/fix up IO::Path.accessed
1cd7de0 Fix up type graph
56256d0 Minor formatting improvements in IO::Special
184342c Document IO::Special.what
6bd0f98 Dissuade readers from using IO::Spec*
7afd9c4 Remove unrelated related classes
a43ecb9 Document IO::Path's $.SPEC and $.CWD
e9b6809 Document IO::Path::* subclasses
9102b51 Fix up IO::Path.basename
5c1d3b6 Document IO::Spec::Unix.basename
a1cb80b Document IO::Spec::Win32.basename
28b6283 Document IO::Spec::*.canonpath
50e5565 Document IO::Spec::*.catdir and .catfile
dbdc995 Document IO::Spec::*.catpath
0ca2295 Reword/expand IO::Path intro prose
45e84ad Move IO::Path.path to attributes
b9de84f Remove DateTime tutorial from IO::Path docs
69b2082 Document IO::Path.chdir
d436f3c Document IO::Spec::* don't do any validation
4090446 Improve chmod docs
1527d32 Document :completely arg to IO::Path.resolve
372545c Straighten up file test docs
a30fae6 Avoid potential confusion with use of word "object"
2aa3c9f Document new behaviour of IO::Handle.Supply
56b50fe Document IO::Handle.slurp
017acd4 Improve docs for IO::Path.slurp
0f49bb5 List Rakudo-supported encodings in open()
e60da5c List utf-* alias examples too since they're common
f83f78c Use idiomatic Perl 6 in example
fff866f Fix docs for symlink/link routines
aeeec94 Straighten up copy, move, rename
923ea05 Straighten up mkdir docs
47b0526 Explicitly spell out caveats of IO::Path.Str
60b9227 Change return value formkdir``
8d95371 Expand IO::Handle/IO::Pipe.path docs
fd8a5ed Document IO::Pipe.path
bd4fa68 Document IO::Handle/IO::Pipe.IO
2aaf12a Document IO::Handle.Str
53f2b99 Documentrole IO's new purpose
160c6a2 Document IO::Handle.lock/.unlock
3145979 Document IO::Path.child-secure
c5524ef Rename IO::Path.concat-with to .add
81a5806 Amend IO::Path.resolve: :completely
6ca67e4 Start sketching out Definitive IO Guide™
b9c9117 Toss IO::Path.child-secure
61cb776 Document IO::Path.sibling
0fc39a6 Fix typegraph
9a63dc4 Document IO::Path.cleanup
2387ce3 Re-write IO::Handle.close docs
0def0d1 Amend IO::Handle.close docs
c7e32e2 Document IO::Spec::Unix.curupdir
fe489dc Document IO::Spec::Unix.curdir
83d5de0 Document IO::Spec::Unix.updir
4804128 Document IO::Handle.DESTROY
c991862 Add warning to dir about...
eca21ff Document copy/move behaviour for same target/source
6c2b8b2 Document IO::Path/IO::Handle.comb
fb29e04 Include exception used in IO::Path.resolve
69d473f Document IO::Spec::*.devnull
994d671 List IO::Dir as one of the means...
4432ef3 Finish up IO::Path.dir docs
64355c8 Document IO::Spec::*.dir-sep
914c100 Finish up IO::Path.dirname
8d5e31c Document IO::Handle.encoding
d5c36aa Finish off IO::Handle.eof
e9de97e Document IO::Spec::*.extension
bf7ec00 Document IO::Handle.flush
25bce38 Document IO::Path.succ
8233960 Improve IO::Handle.t docs
b4006a2 Be explicit what IO::Handle.opened returns
c4f27a7 Document IO::Path.pred
860333f Remove entirely-invented "File test operators"
ab0bd7a Document IO::Path.Numeric/.Int
4f81f08 Improve IO::Handle.get docs
c45d389 Finish off IO::Handle.getc/&getc docs
a4012e0 Document IO::Handle.gist
d15b0c7 Document IO::Path.gist
1cf6932 Document IO::Spec::*.is-absolute
4e88b84 Finish up IO::Path.is-absolute
497e7f7 Finish off IO::Path.is-relative
f7e75c1 Document IO::Handle.nl-in
e309ddd Finish up &note
81900cb Finish off IO::Path.parent
59cbc38 Finish off IO::Path.parts
b99a666 Finish off IO::Path.path/.IO
b070999 Document IO::Spec::*.path
bace8ff Document IO::Path*.perl
dfdd845 Add "The Basics" section to TDIOG
cdc701e Add "What's an IO::Path Anyway?" section to TDIOG
0d6d058 Add "Writing into files" Section to TDIOG
a6365f3 Document IO::Handle.words/&words
2e25c82 Document IO::Spec::*.join
49e58bd Document IO::Handle.lines
1744820 Document IO::Path.lines
f3f70a0 Document IO::Path.words
509f0e8 Fix incorrect suggested routine
a6f1cbf Fix up IO::Handle.print
8f53830 Fix up IO::Handle.print-nl
dc50211 Fix &prompt
98965b3 Fix up IO::Handle.split
bd702e2 Fix up IO::Handle.comb
6dd92b8 Document IO::CatHandle
edeb069 Document IO::Path.split
2d96596 Document IO::Spec::*.split
129c097 Document IO::Spec::*.splitdir
b946960 Document IO::Spec::*.splitpath
dcd7490 Fix rmdir docs
2a7bd17 Document IO::Spec::*.rel2abs
f45241f Document IO::Spec::*.rootdir
70a80ec Document IO::Handle.put
6f58ed0 Polish IO::Handle.say
3790a0f Polish &put/&print/&say
ebb6f53 Document IO::Handle.nl-out attribute
53c9c91 Document IO::Handle.chomp attribute
ca2a3a0 Improve &open/IO::Handle.open docs
856e846 Add Reading From Files section to TDIOG

1 comment

Perl 6 IO TPF Grant: Monthly Report (April, 2017)

By Zoffix Znet on April 19, 2017 4:54 PM

This document is the April, 2017 progress report for TPF Standardization, Test Coverage, and Documentation of Perl 6 I/O Routines grant

Timing

As proposed to and approved by the Grant Manager, I've extended the due date for this grant by 1 extra month, in exchange for doing some extra optimization work on IO routines at no extra cost. The new completion date is May 22nd; right after the next Rakudo compiler release.

Communications

I've created and published three notices as part of this grant, informing the users on what is changing and how to best upgrade their code, where needed:

IO Action Plan Progress

Most of the IO Action Plan has been implemented and got shipped in Rakudo's 2017.04.2 release. The remaining items are:

Implement better way to signal closed handle status (was omited from release due to original suggestion to do this not being ideal; likely better to do this on the VM end)
Implement IO::CatHandle as a generalized IO::ArgFiles (was omited from release because it was decided to make it mostly-usable wherever IO::Handle can be used, and IO::ArgFiles is far from that, having only a handful of methods implemented)
Optimization of the way we perform stat calls for multiple file tests (entirely internal that requires no changes to users' code)

Documentation and Coverage Progress

In my spreadsheet with all the IO routines and their candidates, the totals show that 40% have been documented and tested. Some of the remaining 60% may already have tests/docs added when implementing IO Action Plan or ealier and merely need checking and verification.

Optimizations

Some of the optimizations I promised to deliver in exchange for grant deadline extension were already done on IO::Spec::Unix and IO::Path routines and have made it into the 2017.04.2 release. Most of the optimizations that will be done in the upcoming month will be done in IO::Spec::Win32 and will largely affect Windows users.

IO Optimizations in 2017.04.2 Done by Other Core Members:

Elizabeth Mattijsen made .slurp 2x faster rakudo/b4d80c0
Samantha McVey made nqp::index—which is used in path operations—2x faster rakudo/f1fc879
IO::Pipe.lines was made 3.2x faster by relying on work done by Elizabeth Mattijsen rakudo/0c62815

Tickets Resolved

The following tickets have been resolved as part of the grant:

RT#130460 Can we relax indir's test on the target directory?: Resolved by making indir's default test to be only :d
RT#130900: nul in pathname: Resolved by checking the path for nul and throwing when found
RT#127407: [RFC] (1) add method IO::Path.stemname; (2) expand method IO::Path.parts: RFC rejected, but the described functionality is now available via the .extension method that was made much more powerful as part of IO Action Plan

Possibly more tickets were addressed by the IO Action Plan implementation, but they still need further review.

Bugs Fixed

Fixed a bug in IO::Path.resolve with combiners tucked on the path separator. Fix in rakudo/9d8e391f3b; tests in roast/92217f75ce. The bug was identified by Timo Paulssen while testing secure implementation of IO::Path.child

IO Bug Fixes in 2017.04.2 Done by Other Core Members:

Timo Paulssen fixed a bug with IO::Path types not being accepted by is native NativeCall trait rakudo/99840804
Elizabeth Mattijsen fixed an issue in assignment to dynamics. This made it possible to temp $*TMPDIR variable rakudo/1b9d53
Jonathan Worthington fixed a crash when slurping large files in binary mode with &slurp or IO::Path.slurp rakudo/d0924f1a2
Jonathan Worthington fixed a bug with binary slurp reading zero bytes when another thread is causing a lot of GC rakudo/756877e

Commits

So far, I've commited 192 IO grant commits to rakudo/roast/doc repos.

Rakudo

69 IO grant commits:

c6fd736 Make IO::Spec::Win32.is-absolute about 63x faster
7112a08 Add :D on invocant for file tests
8bacad8 Implement IO::Path.sibling
a98b285 Remove IO::Path.child-secure
0b5a41b Rename IO::Path.concat-with to .add
9d8e391 Fix IO::Path.resolve with combiners; timotimo++
1887114 Implement IO::Path.child-secure
b8458d3 Reword method child for cleaner code
51e4629 Amend rules for last part in IO::Path.resolve
9a2446c Move Bool return value to signature
214198b Implement proper args for IO::Handle.lock
c95c4a7 Make IO::Path/IO::Special do IO role
fd503f8 grant] Remove role IO and its .umask method"
0e36bb2 Make IO::Spec::Win32!canon-cat 2.3x faster
40217ed Swap .child to .concat-with in all the guts
490ffd1 Do not use self.Str in IO::Path errors
1f689a9 Fix up IO::Handle.Str
c01ebea Make IO::Path.mkdir return invocant on success
d46e8df Add IO::Pipe .path and .IO methods
6ee71c2 Coerce mode in IO::Path.mkdir to Int
0d9ecae Remove multi-dir &mkdir
ff97083 Straighten up rename, move, and copy
7f73f92 Make IO::Path.new-from-absolute-path private
da1dea2 Fix &symlink and &link
8c09c84 Fix symlink and link routines
90da80f Rework read methods in IO::Path/IO::Handle
c13480c IO::Path.slurp: make 12%-35% faster; propagate Failures
f1b4af7 Implement IO::Handle.slurp
184d499 Make IO::Handle.Supply respect handle's mode
b6838ee Remove .f check in .z
6a8d63d Implement :completely param in IO::Path.resolve
e681498 Make IO::Path throw when path contains NUL byte
b4358af Delete code for IO::Spec::Win32.catfile
0a442ce Remove type constraint in IO::Spec::Cygwin.canonpath
0c8bef5 Implement :parent in IO::Spec::Cygwin.canonpath
a0b82ed Make IO::Path::* actually instantiate a subclass
67f06b2 Run S32-io/io-special.t test file
954e69e Fix return value of IO::Special methods
a432b3d Remove IO::Path.abspath (part 2)
94a6909 Clean up IO::Spec::Unix.abs2rel a bit
966a7e3 Implement IO::Path.concat-with
50aea2b Restore IO::Handle.IO
15a25da Fix ambiguity in empty extension vs no extension
b1e7a01 Implement IO::Path.extension 2.0
b62d1a7 Give $*TMPDIR a container
099512b Clean up and improve all spurt routines
cb27bce Clean up &open and IO::Path.open
4c31903 Add S32-io/chdir-process.t to list of test files to run
5464b82 Improve &*chdir
2483d68 Fix regression in &chdir's failure mode
ca1acb7 Fix race in &indir(IO::Path …)
a0ef2ed Improve &chdir, &indir, and IO::Path.chdir
aa62cd5 Remove &tmpdir and &homedir
a5800a1 Implement IO::Handle.spurt
36ad92a Remove 15 methods from IO::Handle
87987c2 Remove role IO and its .umask method
9d8d7b2 Log all changes to plan made during review period
0c7e4a0 Do not capture args in .IO method
c360ac2 Fix smartmatch of Cool ~~ IO::Path
fa9aa47 Make R::I::SET_LINE_ENDING_ON_HANDLE 4.1x Faster
0111f10 Make IO::Spec::Unix.catdir 3.9x Faster
4fdebc9 Make IO::Spec::Unix.split 36x Faster
dcf1bb2 Make IO::Spec::Unix.rel2abs 35% faster
55abc6d Improve IO::Path.child perf on *nix
4032953 Make IO::Handle.open 75% faster
a01d679 Remove IO::Path.pipe
212cc8a Remove IO::Path.Bridge
76f7187 Do not cache IO::Path.e results
dd4dfb1 Fix crash in IO::Special .WHICH/.Str

Perl 6 Specification

47 IO grant commits:

3b36d4d Test IO::Path.sibling
7a063b5 Fudge .child-secure tests
39677c4 IO::Path.concat-with got renamed to .add
92217f7 Test IO::Path.child-secure with combiners
f3c5dae Test IO::Path.child-secure
a716962 Amend rules for last part in IO::Path.resolve
4194755 Test IO::Handle.lock/.unlock
64ff572 Cover IO::Path/IO::Pipe's .Str/.path/.IO
637500d Spec IO::Pipe.path/.IO returns IO::Path type object
8fa49e1 Test link routines
416b746 Test symlink routines
d4353b6 Rewrite .l on broken symlinks test
7e4a2ae Swap .slurp-rest to .slurp
a4c53b0 Use bin IO::Handle to test its .Supply
feecaf0 Expand file tests
a809f0f Expand IO::Path.resolve tests
ee7f05b Move is-path sub to top so it can be reused
b16fbd3 Add tests to check nul byte is rejected
8f73ad8 Change \0 roundtrip test to \t roundtrip test
7c7fbb4 Cover :parent arg in IO::Spec::Cygwin.canonpath
896033a Cover IO::Spec::QNX.canonpath
c3c51ed Cover IO::Spec::Win32.basename
d8707e7 Cover IO::Spec::Unix.basename
bd8d167 Test IO::Path::* instantiate a subclass
43ec543 Cover methods of IO::Special
e5dc376 Expand IO::Path.accessed tests
0e47f25 Test IO::Path.concat-with
305f206 Test empty-string extensions in IO::Path.extension
2f09f18 Fix incorrect test
b23e53e Test IO::Path.extension
1d4e881 Test $*TMPDIR can be temped
79ff022 Expand &spurt and IO::Path.spurt tests
ba3e7be Merge S32-io/path.t and S32-io/io-path.t
3c4e81b Test IO::Path.Str works as advertised
86c5f9c Delete qp{} tests
430ab89 Test &*chdir
86f79ce Expand &chdir tests
73a5448 Remove two fudged &chdir tests
04333b3 Test &indir fails with non-existent paths by default
bd46836 Amend &indir race tests
f48198f Test &indir
5a7a365 Expand IO::Spec::*.tmpdir tests
14b6844 Use Numeric instead of IO role in dispatch test
8d6ca7a Cover IO::Path.ACCEPTS
091931a Expand &open tests
465795c Test IO::Path.lines(*) does not crash
63370fe Test IO::Special .WHICH/.Str do not crash

Documentation

76 IO grant commits:

61cb776 Document IO::Path.sibling
b9c9117 Toss IO::Path.child-secure
6ca67e4 Start sketching out Definitive IO Guide™
81a5806 Amend IO::Path.resolve: :completely
c5524ef Rename IO::Path.concat-with to .add
3145979 Document IO::Path.child-secure
160c6a2 Document IO::Handle.lock/.unlock
53f2b99 Document role IO's new purpose
2aaf12a Document IO::Handle.Str
bd4fa68 Document IO::Handle/IO::Pipe.IO
fd8a5ed Document IO::Pipe.path
8d95371 Expand IO::Handle/IO::Pipe.path docs
60b9227 Change return value for mkdir
47b0526 Explicitly spell out caveats of IO::Path.Str
923ea05 Straighten up mkdir docs
aeeec94 Straighten up copy, move, rename
fff866f Fix docs for symlink/link routines
f83f78c Use idiomatic Perl 6 in example
e60da5c List utf-* alias examples too since they're common
0f49bb5 List Rakudo-supported encodings in open()
017acd4 Improve docs for IO::Path.slurp
56b50fe Document IO::Handle.slurp
2aa3c9f Document new behaviour of IO::Handle.Supply
a30fae6 Avoid potential confusion with use of word "object"
372545c Straighten up file test docs
1527d32 Document :completely arg to IO::Path.resolve
4090446 Improve chmod docs
d436f3c Document IO::Spec::* don't do any validation
69b2082 Document IO::Path.chdir
b9de84f Remove DateTime tutorial from IO::Path docs
45e84ad Move IO::Path.path to attributes
0ca2295 Reword/expand IO::Path intro prose
dbdc995 Document IO::Spec::*.catpath
50e5565 Document IO::Spec::*.catdir and .catfile
28b6283 Document IO::Spec::*.canonpath
a1cb80b Document IO::Spec::Win32.basename
5c1d3b6 Document IO::Spec::Unix.basename
9102b51 Fix up IO::Path.basename
e9b6809 Document IO::Path::* subclasses
a43ecb9 Document IO::Path's $.SPEC and $.CWD
7afd9c4 Remove unrelated related classes
6bd0f98 Dissuade readers from using IO::Spec*
184342c Document IO::Special.what
56256d0 Minor formatting improvements in IO::Special
1cd7de0 Fix up type graph
b3a9324 Expand/fix up IO::Path.accessed
1973010 Document IO::Path.ACCEPTS
cc62dd2 Kill IO::Path.abspath
1f75ddc Document IO::Spec*.abs2rel
24a6ea9 Toss all of the TODO methods in IO::Spec*
65cc372 Document IO::Path.concat-with
b9e692e Document new IO::Path.extension
d5abceb Write docs for all spurt routines
b8fba97 Point out my $*CWD = chdir … is an error
b78d4fd Include type names in links to methods
bdd18f1 Fix desc of IO::Path.Str
a53015a Clarify value of IO::Path.path
5aa614f Improve suggestion for Perl 5's opendir
bf377c7 Document &indir
e5225be Fix URL to &*chdir
e1a299c Reword "defined as" for &*chdir
3fdc6dc Document &*chdir
1d0e433 Document &chdir
d050d4b Remove IO::Path.chdir prose
839a6b3 Expand docs for $*HOME and $*TMPDIR
db36655 Remove tip to use $*SPEC to detect OS
0511e07 Document IO::Spec::*.tmpdir
cc6539b Remove 8 methods from IO::Handle
335a98d Remove mention of role IO
cc496eb Remove mention of IO.umask
3cf943d Expand IO::Path.relative
ccae74a Fix incorrect information for IO::Path.absolute
d02ae7d Remove IO::Handle.rw and .rwx
69d32da Remove IO::Handle.z
110efb4 No need for .ends-with
fd7a41b Improve code example

0 comments

Perl 5 Porters Mailing List Summary: September 5th-11th

By Sawyer X on September 12, 2016 4:34 PM

Hey everyone,

Following is the p5p (Perl 5 Porters) mailing list summary for the past week. Enjoy!

0 comments

Hacking on The Rakudo Perl 6 Compiler: Mix Your Fix

By Zoffix Znet on August 1, 2016 8:40 AM under Perl 6, Perl 6, perl6

Read this article on Perl6.Party

While testing a fix for one of the Less Than Awesome behaviours in standalone Signature objects, I came across a bugglet. Smartmatching two Signatures throws, while spilling a bit of the guts:

<Zoffix> m: my $m = method ($a: $b) { }; say $m.signature ~~ :($a, $b);
<camelia> rakudo-moar 46838d: OUTPUT«Method 'type' not found for invocant of class 'Any'␤ in block at line 1␤␤»

So I figured I'll write about fixing it, 'cause hacking on internals is lots of fun. Let's roll!

Golf It Down

The less code there is to reproduces the bug, the fewer places there are for that bug to hide. We have a detached method and then we smartmatch its signature against something else. Let's try to golf it down a bit and smartmatch two Signatures, without involving a method:

<Zoffix> m: :($a, $b) ~~ :($a, $b);
<camelia> rakudo-moar 46838d: ( no output )

The bug disappeared, so perhaps out Signature on the left doesn't contain the stuff that triggers the bug. Let's dump the signature of the method to see what we should match against:

<Zoffix> m: my $m = method ($a: $b) { }; say $m.signature <camelia> rakudo-moar 46838d: OUTPUT«($a: $b, *%_)␤»

Aha! It has a slurpy hash: *%_. Let's try matching a Signature with a slurpy in it:

<Zoffix> m: :(*%) ~~ :();
<camelia> rakudo-moar 46838d: OUTPUT«Method 'type' not found for invocant of class 'Any'␤ in block at line 1␤␤»

And there we go: hole in three. Let's proceed.

Roast It

There's an official Perl 6 test suite that Rakudo must pass to be called a Perl 6 compiler. Since we got a bug on our hands, we should add a test for it to the test suite to ensure it doesn't rear its ugly head again.

The copy of the repo gets automatically cloned into t/spec when you run make spectest in Rakudo's checkout. If you don't have a commit bit, you can just change the remote/branch of that checkout to your fork:

cd t/spec
git remote rm origin
git remote add origin https://github.com/YOURUSERNAME/roast
git checkout your-branch
cd ../..

It may be tricky to figure out which file to put the test in, if you're new. You can always ask the good folks on irc.freenode.net/#perl6 for advice. In this case, I'll place the test into S06-signature/outside-subroutine.t

While not required, I find it helpful to open a ticket for the bug. This way I can reference it in my fix in the compiler repo, I can reference it in the commit to the test repo, and people get a place where to tell me why I'm being stupid when I am. I opened this bug as RT#128795.

Now, for the code of the test itself. I'll adjust the plan at the top of the file to include however many tests I'm writing—in this case one. I'll use the lives-ok test sub and stick our buggy golfed code into it. Here's the diff of the changes to the file; note the reference to the ticket number in the comment before the test:

@@ -1,7 +1,7 @@
  use v6;
  use Test;

 -plan 3;
 +plan 4;

  # RT #82946
  subtest 'signature binding outside of routine calls' => {
 @@ -25,4 +25,7 @@ subtest 'smartmatch on signatures with literal strings' => {
  # RT #128783
  lives-ok { EVAL ’:($:)‘ }, ’signature marker is allowed in bare signature‘;

 +# RT #128795
 +lives-ok { :(*%)~~ :() }, 'smartmatch with no slurpy on right side';
 +
  # vim: ft=perl6

Run the file now to ensure the test fails. Hint: some files have fudging; explaining it is out of the scope of this article, but if you notice failures you're not expecting, look it up.

$ make t/spec/S06-signature/outside-subroutine.t
...
Test Summary Report
-------------------
t/spec/S06-signature/outside-subroutine.t (Wstat: 256 Tests: 4 Failed: 1)
  Failed test:  4
  Non-zero exit status: 1

With the test in place, it's time to look at some source code. Let the bug hunt begin!

Make it Saucy

Our bug involves a Smartmatch operator, which aliases the left side to the topic variable $_ and calls .ACCEPTS method on the right side with it. Both of our sides are Signature objects, so let's pop open Rakudo's sauce code for that class.

In the Rakudo's repo, directory src/core/ contains most of the built in types in separate files named after those types, so we'll just pop open src/core/Signature.pm in the editor and locate the definition of method ACCEPTS.

There are actually four multis for ACCEPTS. Here's the full code. Don't try to understand all of it, just note its size.

``` multi method ACCEPTS(Signature:D: Capture $topic) { nqp::p6bool(nqp::p6isbindable(self, nqp::decont($topic))); }

multi method ACCEPTS(Signature:D: @topic) {
    self.ACCEPTS(@topic.Capture)
}

multi method ACCEPTS(Signature:D: %topic) {
    self.ACCEPTS(%topic.Capture)
}

multi method ACCEPTS(Signature:D: Signature:D $topic) {
    my $sclass = self.params.classify({.named});
    my $tclass = $topic.params.classify({.named});
    my @spos := $sclass{False} // ();
    my @tpos := $tclass{False} // ();

    while @spos {
        my $s;
        my $t;
        last unless @tpos && ($t = @tpos.shift);
        $s=@spos.shift;
        if $s.slurpy or $s.capture {
            @spos=();
            @tpos=();
            last;
        }
        if $t.slurpy or $t.capture {
            return False unless any(@spos) ~~ {.slurpy or .capture};
            @spos=();
            @tpos=();
            last;
        }
        if not $s.optional {
            return False if $t.optional
        }
        return False unless $t ~~ $s;
    }
    return False if @tpos;
    if @spos {
        return False unless @spos[0].optional or @spos[0].slurpy or @spos[0].capture;
    }

    for flat ($sclass{True} // ()).grep({!.optional and !.slurpy}) -> $this {
        my $other;
        return False unless $other=($tclass{True} // ()).grep(
            {!.optional and $_ ~~ $this });
        return False unless +$other == 1;
    }

    my $here=($sclass{True}:v).SetHash;
    my $hasslurpy=($sclass{True} // ()).grep({.slurpy});
    $here{@$hasslurpy} :delete;
    $hasslurpy .= Bool;
    for flat @($tclass{True} // ()) -> $other {
        my $this;

        if $other.slurpy {
            return False if any($here.keys) ~~ -> Any $_ { !(.type =:= Mu) };
            return $hasslurpy;
        }
        if $this=$here.keys.grep( -> $t { $other ~~ $t }) {
            $here{$this[0]} :delete;
        }
        else {
            return False unless $hasslurpy;
        }
    }
    return False unless self.returns =:= $topic.returns;
    True;
}

```

The error we get from the bug mentions .type method call and there is one such method call in the code above (close to the end of it). In this case, there's quite a bit of code to sort through. It would be nice to be able to play around with it, stick a couple of dd or say calls to dump out variables, right?

That approach, however, is somewhat annoying because after each change we have to recompile the entire Rakudo. On the meatiest box I got, it takes about 60 seconds. Not the end of the world, but there's a way to make things lightning fast!

Mix Your Fix

We need to fix a bug in a method of a class. Another way to think of it is: we need to replace a broken method with a working one. Signature class is just like any other class, so if we want to replace one of its methods, we can just mix in a role!

The broken ACCEPTS will continue to live in the compiler, and we'll pop open a separate playground file and define a role—let's calls it FixedSignature—in it. To get our new-and-improved ACCEPTS method in standalone signature objects, we'll use the but operator to mix the FixedSignature in.

Here's the role, the mixing in, and the code that triggers the bug. I'll leave out method bodies for brieviety, but there's they are the same as in the code above.

role FixedSignature {
    multi method ACCEPTS(Signature:D: Capture $topic)     { #`(redacted for brevity) }
    multi method ACCEPTS(Signature:D: @topic)             { #`(redacted for brevity) }
    multi method ACCEPTS(Signature:D: %topic)             { #`(redacted for brevity) }
    multi method ACCEPTS(Signature:D: Signature:D $topic) { #`(redacted for brevity) }
}

my $a = :(*%) but FixedSignature;
my $b = :()   but FixedSignature;

say $a ~~ $b;

There are two more things we need to do for our role to work properly. First, we're dealing with multis and right now the multis in our role are creating ambiguities with the multis in the original Signature class. To avoid that, we'll define a proto:

proto method ACCEPTS (|) { * }

Since the code is using some NQP, we also need to bring in those features into our playground file with the role. Just add the appropriate pragma at the top of the file:

use MONKEY-GUTS;

With these modifications, our final test file becomes the following:

use MONKEY-GUTS;

role FixedSignature {
    proto method ACCEPTS (|) { * }

    multi method ACCEPTS(Signature:D: Capture $topic)     { #`(redacted for brevity) }
    multi method ACCEPTS(Signature:D: @topic)             { #`(redacted for brevity) }
    multi method ACCEPTS(Signature:D: %topic)             { #`(redacted for brevity) }
    multi method ACCEPTS(Signature:D: Signature:D $topic) { #`(redacted for brevity) }
}

my $a = :(*%) but FixedSignature;
my $b = :()   but FixedSignature;

say $a ~~ $b;

And with this trick in place, we now have a rapid-fire weapon to hunt down the bug with—the changes we make compile instantly.

Pull The Trigger

Now, we can debug the code just like any other. I prefer applying liberal amounts of dd (or say) calls and dumping out the variables to ensure their contents match expectations.

The .type method call our error message mentions is in this line:

return False if any($here.keys) ~~ -> Any $_ { !(.type =:= Mu) };

It calls it on the keys of $here, so let's dump the $here before that statement:

...
dd $here
return False if any($here.keys) ~~ -> Any $_ { !(.type =:= Mu) };
...
# OUTPUT:
# SetHash $here = SetHash.new(Any)

Here's our offending Any, let's go up a bit and dump the $here right where it's defined:

...
my $here=$sclass{True}.SetHash;
dd $here;
...
# OUTPUT:
# SetHash $here = SetHash.new(Any)

It's still there, and for a good reason. If we trace the creation of $sclass, we'll see it's this:

my $sclass = self.params.classify({.named});

The params of the Signature on the right of the smartmatch get classified based on whether they are named or not. The named parameters will be inside a list under the True key of $sclass. Since we do not have any named params, there won't be such a key, and we can verify that with this bit of code:

:().params.classify(*.named).say
# OUTPUT:
# {}

When we go to define $here, we get an Any from $sclass{True}, since that key doesn't exist, and when we call .SetHash on it, we get our problematic Sethash object with an Any in it. And so, we have our fix for the bug: ensure the True key in $sclass is actually there before creating a SetHash out of its value:

my $here=($sclass{True}:v).SetHash;

Add that to our playground file with the FixedSignature role in it, run it, and verify the fix works. Now, simply transplant the fix back into src/core/Signature.pm and then compile the compiler.

perl Configure.pl --gen-moar --gen-nqp --backends=moar
make
make test
make install

Verify our fix worked before we proceed onto the final stages:

$ make t/spec/S06-signature/outside-subroutine.t
...
All tests successful.
Files=1, Tests=4,  1 wallclock secs ( 0.03 usr  0.00 sys +  0.32 cusr  0.02 csys =  0.37 CPU)
Result: PASS

A Clean Kill

So far, all we know is the bug we found was fixed and the tests we wrote for it pass. However, before we ship our fix, we must ensure we didn't break anything else. There are other devs working from the same repo and you'll be interfering with their work if you break stuff.

Run the full Roast test suite with make spectest command. You can use the TEST_JOBS environmental variable to specify the number of simultaneous tests. Generally a value slightly higher than the available cores works the fastest... and cores make all the difference. On my 24-core VM I cut releases on, the spectest completes in about 1 minute and 15 seconds. On my 2-core web server, it takes about 25 minutes. You get the idea.

TEST_JOBS=28 make spectest
...
All tests successful.
Files=1111, Tests=52510, 82 wallclock secs (13.09 usr 2.44 sys + 1517.34 cusr 97.67 csys = 1630.54 CPU)
Result: PASS

Once the spectest completes and we have the clean bill of health, we're ready to ship our fix. Commit the Rakudo fix, then go into t/spec and commit the Roast fix:

git commit -m 'Fix Smartmatch with two signatures, only one of which has slurpy hash' \
           -m 'Fixes RT#128795' src/core/Signature.pm
git push

cd t/spec
git commit -m 'smartmatch on signature with no slurpy on right side does not crash' \
           -m 'RT#128795' S06-signature/outside-subroutine.t
git push

If you're pushing to your fork of these projects, you have to go the extra step and submit a Pull Request (just go to your fork and GitHub should display a button just for that).

And we're done! Celebrate with the appropriate amount of fun.

Conclusion

Rakudo bugs can be easy to fix, requiring not much more than knowledge of Perl 6. To fix them, you don't need to re-compile the entire compiler, but can instead define a small role with a method you're trying to fix and modify and recompile just that.

It's important to add tests for the bug into the official test suite and it's also important to run the full spectest after you fix the bug. But most important of all, is to have fun fixing it.

-Ofun

0 comments

IRC::Client: Perl 6 Multi-Server IRC (or Awesome Async Interfaces with Perl 6)

By Zoffix Znet on July 30, 2016 1:06 AM under Perl 6

Read this article on Perl6.Party

I wrote my first Perl 6 program—a New Years IRC Party bot—around Christmas, 2015. The work included releasing the IRC::Client module, and given my virginity with the language and blood alcohol level appropriate for the Holiday Season, the module ended up sufficiently craptastic.

Recently, I needed a tool for some Perl 6 bug queue work, so I decided to lock myself up for a weekend and re-design and re-write the module from scratch. Multiple people bugged me to do so over the past months, so I figured I'd also write a tutorial for how to use the module—as an apology for being a master procrastinator. And should IRC be of no interest to you, I hope the tutorial will prove useful as a general example of async, non-blocking interfaces in Perl 6.

The Basics

To create an IRC bot, instantiate an IRC::Client object, giving it some basic info, and call the .run method. Implement all of the functionality you need as classes with method names matching the events you want to listen to and hand those in via the .plugins attribute. When an IRC event occurs, it's passed to all of the plugins, in the order you specify them, stopping if a plugin claims it handled the event.

Here's a simple IRC bot that responds to being addressed in-channel, notices, and private messages sent to it. The response is the uppercased original message the bot received:

use IRC::Client;
.run with IRC::Client.new:
    :nick<MahBot>
    :host<irc.freenode.net>
    :channels<#perl6>
    :debug
    :plugins(class { method irc-to-me ($_) { .text.uc } })

And here's what the bot looks like when running:

<Zoffix> MahBot, I ♥ you!
<MahBot> Zoffix, I ♥ YOU!

The :nick, :host, and :channels are the nick for your bot, the server it should connect to, and channels it should join. The :debug controls how much debugging output to display. We'll set it to value 1 here, for sparse debug output, just to see what's happening. Tip: install the optional Terminal::ANSIColor module to make debug output purty:

For the .plugins attribute, we hand in an anonymous class. If you have multiple plugins, just shove them all in in the order you want them to receive events in:

:plugins(PlugFirst.new, PlugSecond.new(:conf), class { ... })

The plugin class of our uppercasing bot has a single method that listens to irc-to-me event, triggered whenever the bot is addressed in-channel or is sent a private message or notice. It receives a single argument: one of the objects that does the IRC::Client::Message role. We stick it into the $_ topical variable to save a bit of typing.

We reply to the event by returning a value from the method. The original text is contained inside the .text attribute of the message object, so we'll call .uc method on it to uppercase the content and that's what our reply will be.

As awesome as our uppercasing bot is, it's as useful as an air conditioner on a polar expedition. Let's teach it some tricks.

Getting Smarter

We'll call our new plugin Trickster and it'll respond to commands time—that will give the local time and date—and temp—that will convert temperature between Fahrenheit and Celsius. Here's the code:

use IRC::Client;

class Trickster {
    method irc-to-me ($_) {
        given .text {
            when /time/ { DateTime.now }
            when /temp \s+ $<temp>=\d+ $<unit>=[F|C]/ {
                when $<unit> eq 'F' { "That's {($<temp> - 32) × .5556}°C" }
                default             { "That's { $<temp> × 1.8 + 32   }°F" }
            }
            'huh?'
        }
    }
}

.run with IRC::Client.new:
    :nick<MahBot>
    :host<irc.freenode.net>
    :channels<#perl6>
    :debug
    :plugins(Trickster)

<Zoffix> MahBot, time
<MahBot> Zoffix, 2016-07-23T19:00:15.795551-04:00
<Zoffix> MahBot, temp 42F
<MahBot> Zoffix, That's 5.556°C
<Zoffix> MahBot, temp 42C
<MahBot> Zoffix, That's 107.6°F
<Zoffix> MahBot, I ♥ you!
<MahBot> Zoffix, huh?

The code is trivial: we pass the given text over a couple of regexes. If it contains word time, we return the current time. If it contains word temp we do the appropriate math, based on whether the given number is postfixed by an F or a C. And if no matches happen, we end up returning the inquisitive huh?.

There's an obvious problem with this new and improved plugin: the bot no longer loves me! And while I'll survive the heartache, I doubt any other plugin will teach the bot to love again, as Trickster consumes all irc-to-me events, even if it doesn't recognize any of the commands it can handle. Let's fix that!

Passing The Buck

There's a special value that can be returned by the event handler to signal that it did not handle the event and that it should be propagated to further plugins and event handlers. That value is provided by the .NEXT attribute offered by the IRC::Client::Plugin role, which a plugin does to obtain that attribute. The role is automatically exported when you use IRC::Client.

Let's look at some code utilizing that special value. Note that since .NEXT is an attribute and we can't look up attributes on type objects, you need to go the extra step and instantiate your plugin classes when giving them to :plugins.

use IRC::Client;

class Trickster does IRC::Client::Plugin {
    method irc-to-me ($_) {
        given .text {
            when /time/ { DateTime.now }
            when /temp \s+ $<temp>=\d+ $<unit>=[F|C]/ {
                when $<unit> eq 'F' { "That's {($<temp> - 32) × .5556}°C" }
                default             { "That's { $<temp> × 1.8 + 32   }°F" }
            }
            $.NEXT;
        }
    }
}

class BFF does IRC::Client::Plugin {
    method irc-to-me ($_) {
        when .text ~~ /'♥'/ { 'I ♥ YOU!' };
        $.NEXT;
    }
}

.run with IRC::Client.new:
    :nick<MahBot>
    :host<irc.freenode.net>
    :channels<#perl6>
    :debug
    :plugins(Trickster.new, BFF.new)

<Zoffix> MahBot, time
<MahBot> Zoffix, 2016-07-23T19:37:45.788272-04:00
<Zoffix> MahBot, temp 42F
<MahBot> Zoffix, That's 5.556°C
<Zoffix> MahBot, temp 42C
<MahBot> Zoffix, That's 107.6°F
<Zoffix> MahBot, I ♥ you!
<MahBot> Zoffix, I ♥ YOU!

We now have two plugins that both subscribe to irc-to-me event. The :plugins attribute receives Trickster plugin first, so its event handler will be run first. If the received text does not match either of the Trickster's regexes, it returns $.NEXT from the method.

That signals the Client Object to go hunting for other handlers, so it gets to BFF's irc-to-me handler. There, we reply if the input contains a heart, if not, we pre-emptively return $.NEXT here too.

While the bot got its sunny disposition back, it did so at the cost of quite a bit of extra typing. What can we do about that?

Multify All The Things!

Perl 6 supports multi-dispatch as well as type constraints in signatures. On top of that, smartmatch against IRC::Client's message objects that have a .text attribute uses the value of that attribute. Combine all three of those features and you end up with ridiculously concise code:

use IRC::Client;
class Trickster {
    multi method irc-to-me ($ where /time/) { DateTime.now }
    multi method irc-to-me ($ where /temp \s+ $<temp>=\d+ $<unit>=[F|C]/) {
        $<unit> eq 'F' ?? "That's {($<temp> - 32) × .5556}°C"
                       !! "That's { $<temp> × 1.8 + 32   }°F"
    }
}

class BFF { method irc-to-me ($ where /'♥'/) { 'I ♥ YOU!' } }

.run with IRC::Client.new:
    :nick<MahBot>
    :host<irc.freenode.net>
    :channels<#perl6>
    :debug
    :plugins(Trickster, BFF)

<Zoffix> MahBot, time
<MahBot> Zoffix, 2016-07-23T19:59:44.481553-04:00
<Zoffix> MahBot, temp 42F
<MahBot> Zoffix, That's 5.556°C
<Zoffix> MahBot, temp 42C
<MahBot> Zoffix, That's 107.6°F
<Zoffix> MahBot, I ♥ you!
<MahBot> Zoffix, I ♥ YOU!

Outside of the signature, we no longer have any need for the message object, so we use the anonymous $ parameter in its place. We then type-constrain that parameter with a regex match, and so the method will be called only if the text of the message matches that regex. Since no methods will be called on failed matches, we no longer have to mess around with the whole $.NEXT business or compose any roles into our plugins.

The bodies of our methods each have a single statement that produces the response value for the event. In the temperature converter, we use the ternary operator to select which formula to use for the conversion, depending on the unit requested, and yes, the $<unit> and $<temp> captures created in the signature type constraint match are available in the method's body.

An Eventful Day

Along with standard named and numerical IRC protocol events, IRC::Client offers convenience events. One of them we've already seen: the irc-to-me event. Such events are layered, so one IRC event can trigger several IRC::Client's events. For example, if someone addresses our bot in a channel, the following chain of events will be fired:

irc-addressed  ▶  irc-to-me  ▶  irc-privmsg-channel  ▶  irc-privmsg  ▶  irc-all

The events are ordered from "narrowest" to "widest": irc-addressed can be triggered only in-channel, when our bot is addressed; irc-to-me can also be triggered via notice and private message, so it's wider; irc-privmsg-channel includes all channel messages, so it's wider still; and irc-privmsg also includes private messages to our bot. The chain ends by the widest event of them all: irc-all.

If a plugin's event handler returns any value other than $.NEXT, later events in the event chain won't be fired, just as plugins later in the plugin chain won't be tried for the same reason. Each event is tried on all of the plugins, before attempting to handle a wider event.

By setting the :debug attribute to level 3 or higher, you'll get emitted events in the debug output. Here's our bot attempting to handle unknown command blarg and then processing command time handled by irc-to-me event handler we defined:

All of IRC::Client's events have irc- prefix, so you can freely define auxiliary methods in your plugin, without worrying about conflicting with event handlers. Speaking of emitting things...

Keep 'Em Commin'

Responding to commands is sweet and all, but many bots will likely want to generate some output out of their own volition. As an example, let's write a bot that will annoy us whenever we have unread GitHub notifications!

use IRC::Client;
use HTTP::Tinyish;
use JSON::Fast;

class GitHub::Notifications does IRC::Client::Plugin {
    has Str  $.token  = %*ENV<GITHUB_TOKEN>;
    has      $!ua     = HTTP::Tinyish.new;
    constant $API_URL = 'https://api.github.com/notifications';

    method irc-connected ($) {
        start react {
            whenever self!notification.grep(* > 0) -> $num {
                $.irc.send: :where<Zoffix>
                            :text("You have $num unread notifications!")
                            :notice;
            }
        }
    }

    method !notification {
        supply {
            loop {
                my $res = $!ua.get: $API_URL, :headers{ :Authorization("token $!token") };
                $res<success> and emit +grep *.<unread>, |from-json $res<content>;
                sleep $res<headers><X-Poll-Interval> || 60;
            }
        }
    }
}

.run with IRC::Client.new:
    :nick<MahBot>
    :host<irc.freenode.net>
    :channels<#perl6>
    :debug
    :plugins(GitHub::Notifications.new)

[00:25:41] -MahBot- Zoffix, You have 20 unread notifications!
[00:26:41] -MahBot- Zoffix, You have 19 unread notifications!

We create GitHub::Notifications class that does the IRC::Client::Plugin role. That role gives us the $.irc attribute, which is the IRC::Client object we'll use to send messages to us on IRC.

Aside from irc-connected method, the class is just like any other: a public $.token attribute for our GitHub API token, a private $!ua attribute that keeps our HTTP User Agent object around, and a private notification method, where all the action happens.

Inside notification, we create a Supply that will emit the number of unread notifications we have. It does so by using an HTTP::Tinyish object to access a GitHub API endpoint. On line 24, it parses the JSON returned by successful requests, and greps the message list for any items with unread property set to true. The prefix + operator converts the list to an Int that is total items found, which is what we emit from our supply.

The irc-connected event handler gets triggered when we successfully connect to an IRC server. In it, we start an event loop that reacts whenever we receive the current unread messages count from our supply given by notifications method. Since we're only interested in cases where we do have unread messages, we also pop a grep on the supply to filter out the cases without any messages (yes, we could avoid emitting those in the first place, but I'm showing off Perl 6 here 😸). And once we do have unread messages, we simply call IRC::Client's .send method, asking it to send us an IRC notice with the total number of unread messages. Pure awesomeness!

Don't Wait Up

We've covered the cases where we either have an asynchronous supply of values we sent to IRC or where we reply to a command right away. It's not uncommon for a bot command to take some time to execute. In those cases, we don't want the bot to lock up while the command is doing its thing.

Thanks to Perl 6's excellent concurrency primitives, it doesn't have to! If an event handler returns a Promise, the Client Object will use its .result as the reply when it is kept. This means that in order to make our blocking event handler non-blocking, all we have to do is wrap its body in a start { ... } block. What could be simpler?

As an example, let's write a bot that will respond to bash command. The bot will fetch bash.org/?random1, parse out the quotes from the HTML, and keep them in the cache. When the command is triggered, the bot will hand out one of the quotes, repeating the fetching when the cache runs out. In particular, we don't want the bot to block while retrieving and parsing the web page. Here's the full code:

use IRC::Client;
use Mojo::UserAgent:from<Perl5>;

class Bash {
    constant $BASH_URL = 'http://bash.org/?random1';
    constant $cache    = Channel.new;
    has        $!ua    = Mojo::UserAgent.new;

    multi method irc-to-me ($ where /bash/) {
        start $cache.poll or do { self!fetch-quotes; $cache.poll };
    }

    method !fetch-quotes {
        $cache.send: $_
            for $!ua.get($BASH_URL).res.dom.find('.qt').each».all_text.lines.join: '  ';
    }
}

.run with IRC::Client.new:
    :nick<MahBot>
    :host<irc.freenode.net>
    :channels<#perl6>
    :debug
    :plugins(Bash.new)

<Zoffix> MahBot, bash
<MahBot> Zoffix, <Time> that reminds me of when Manning and I installed OS/2 Warp4 on a box and during the install routine it said something to the likes of 'join the hundreds of people on the internet'

For page fetching needs, I chose Perl 5's Mojo::UserAgent, since it has an HTML parser built-in. The :from<Perl5> adverb indicates to the compiler that we want to load a Perl 5, not Perl 6, module.

Since we're multi-threading, we'll use a Channel as a thread-safe queue for our caching purposes. We subscribe to the irc-to-me event where text contains word bash. When the event handler is triggered, we pop out to a new thread using the start keyword. Then we .poll our cache and use the cached value if we have one, otherwise, the logic will move onto the do block that that calls the fetch-quotes private method and when that completes, polls the cache once more, getting a fresh quote. All said and done, a quote will be the result of the Promise we return from the event handler.

The fetch-quotes method fires up our Mojo::UserAgent object that fetches the random quotes page from the website, finds all HTML elements that have class="qt" on them—those are paragraphs with quotes. Then, we use a hyper method call to convert those paragraphs to just text and that final list is fed to our $cache Channel via a for loop. And there you go, we non-blockingly connected our bot to the cesspit of the IRC world. And speaking of things you may want to filter...

Watch Your Mouth!

Our bot would get banned rather quickly if it spewed enormous amounts of output into channels. An obvious solution is to include logic in our plugins that would use a pastebin if the output is too large. However, it's pretty impractical to add such a thing to every plugin we write. Luckily, IRC::Client has support for filters!

For any method that issues a NOTICE or PRIVMSG IRC command, IRC::Client will pass the output through classes given to it via :filters attribute. This means we can set up a filter that will automatically pastebin large output, regardless of what plugin it comes from.

We'll re-use our bash.org quote bot, except this time it will pastebin large quotes to Shadowcat pastebin. Let's look at some code!

use IRC::Client;
use Pastebin::Shadowcat;
use Mojo::UserAgent:from<Perl5>;

class Bash {
    constant $BASH_URL = 'http://bash.org/?random1';
    constant $cache    = Channel.new;
    has        $!ua    = Mojo::UserAgent.new;

    multi method irc-to-me ($ where /bash/) {
        start $cache.poll or do { self!fetch-quotes; $cache.poll };
    }

    method !fetch-quotes {
        $cache.send: $_
            for $!ua.get($BASH_URL).res.dom.find('.qt').each».all_text;
    }
}

.run with IRC::Client.new:
    :nick<MahBot>
    :host<irc.freenode.net>
    :channels<#zofbot>
    :debug
    :plugins(Bash.new)
    :filters(
        -> $text where .lines > 1 || .chars > 300 {
            Pastebin::Shadowcat.new.paste: $text.lines.join: "\n";
        }
    )

<Zoffix> MahBot, bash
<MahBot> Zoffix, <intuit> hmm maybe sumtime next week i will go outside'
<Zoffix> MahBot, bash
<MahBot> Zoffix, http://fpaste.scsys.co.uk/528741

The code that does all the filtering work is small enough that it's easy to miss—it's the last 5 lines in the program above. The :filters attribute takes a list of Callables, and here we're passing a pointy block. In its signature we constraint the text to be more than 1 line or more than 300 characters long, so our filter will be run only when those criteria are met. Inside the block, we simply use the Pastebin::Shadowcat module to throw the output onto the pastebin. Its .paste method returns the URL of the newly-created paste, which is what our filter will replace the original content with. Pretty awesome!

It Spreads Like Butter

In the past, when I used other IRC client tools, whenever someone asked me to place my bots on other servers, the procedure was simple: copy over the code to another directory, change config, and you're done. It almost made sense that a new server would mean a "new" bot: different channels, different nicknames, and so on.

In Perl 6's IRC::Client, I tried to re-imagine things a bit: a server is merely another identifier for a message, along with a channel or nickname. This means connecting your bot to multiple servers is as simple as adding new server configuration via :servers attribute:

use IRC::Client;

class BFF {
    method irc-to-me ($ where /'♥'/) { 'I ♥ YOU!' }
}

.run with IRC::Client.new:
    :debug
    :plugins(BFF)
    :nick<MahBot>
    :channels<#zofbot>
    :servers(
        freenode => %(
            :host<irc.freenode.net>,
        ),
        local => %(
            :nick<P6Bot>,
            :channels<#zofbot #perl6>,
            :host<localhost>,
        )
    )

[on Freenode server]
<ZoffixW> MahBot, I ♥ you
<MahBot> ZoffixW, I ♥ YOU!

[on local server]
<ZoffixW> P6Bot, I ♥ you
<P6Bot> ZoffixW, I ♥ YOU!

First, our plugin remains oblivious that it's being run on multiple servers. Its replies get redirected to the correct server and IRC::Client still executes its method handler in a thread-safe way.

In the IRC::Client's constructor we added :servers attribute that takes a Hash. The keys of this Hash are servers' labels and values are server-specific configurations that override global settings. So freenode server gets its :nick and :channels from the :nick and :channels attributes we give to IRC::Client, while the local server overrides those with its own values.

The debug output now has server lables printed, to indicate to which server the event applies:

And so, but simply telling the bot to connect to another server, we made it multi-server, without making any changes to our plugins. But what do we do when we do want to talk to a specific server?

Send It That Way

When the bot is .run, the Client Object changes the values of :servers attribute to be IRC::Client::Server objects. Those stringify to the label for the server they represent and we can get them either from the .server attribute of the Message Object or .servers hash attribute of the Client Object. Client Object methods such as .send or .join take an optional server attribute that controls which server the message will be sent to and defaults to value *, which means send to every server.

Here's a bot that connects to two servers and joins several channels. Whenever it sees a channel message, it forwards it to all other channels and sends a private message to user Zoffix on server designated by label local.

use IRC::Client;

class Messenger does IRC::Client::Plugin {
    method irc-privmsg-channel ($e) {
        for $.irc.servers.values -> $server {
            for $server.channels -> $channel {
                next if $server eq $e.server and $channel eq $e.channel;

                $.irc.send: :$server, :where($channel), :text(
                    "$e.nick() over at $e.server.host()/$e.channel() says $e.text()"
                );
            }
        }

        $.irc.send: :where<Zoffix>
                    :text('I spread the messages!')
                    :server<local>;
    }
}

.run with IRC::Client.new:
    :debug
    :plugins[Messenger.new]
    :nick<MahBot>
    :channels<#zofbot>
    :servers{
        freenode => %(
            :host<irc.freenode.net>,
        ),
        local => %(
            :nick<P6Bot>,
            :channels<#zofbot #perl6>,
            :host<localhost>,
        )
    }

[on Freenode server/#zofbot]
<ZoffixW> Yey!
[on local server/#zofbot]
<P6Bot> ZoffixW over at irc.freenode.net/#zofbot says Yey!
[on local server/#perl6]
<P6Bot> ZoffixW over at irc.freenode.net/#zofbot says Yey!
[on local server/ZoffixW private message queue]
<P6Bot> I spread the messages!

We subscribe to the irc-privmsg-channel event and when it's triggered, we loop over all the servers. For each server, we loop over all of the connected channels and use $.irc.send method to send a message to that particular channel and server, unless the server and channel are the same as where the message originated.

The message itself calls .nick, .channel, and .server.host methods on the Message Object to identify the sender and origin of the message.

Conclusion

Perl 6 offers powerful concurrency primitives, dispatch methods, and introspection that lets you build awesome non-blocking, event-based interfaces. One of them is IRC::Client that lets you use IRC networks. It's here. It's ready. Use it!

0 comments

Perl 6: The S/// Operator

By Zoffix Znet on April 28, 2016 11:57 PM

Coming from a Perl 5 background, my first experience with Perl 6's non-destructive substitution operator S/// looked something like this:

(artist's impression)

You'll fare better, I'm sure. Not only have the error messages improved, but I'll also explain everything right here and now.

The Smartmatch

The reason I had issues is because, seeing familiar-looking operators, I simply translated Perl 5's binding operator (=~) to Perl 6's smartmatch operator (~~) and expected things to work. The S/// was not documented and, combined with the confusing (at the time) warning message, this was the source of my pain:

my $orig = 'meowmix';
my $new = $orig ~~ S/me/c/;
say $new;

# OUTPUT warning:
# Smartmatch with S/// can never succeed

The old warning suggests the ~~ operator is the wrong choice here and it is. The ~~ isn't the equivalent of Perl 5's =~. It aliases the left hand side to $_, evaluates the right hand side, and then calls .ACCEPTS($_) on it. That is all there is to its magic.

So what's actually happening in the example above:

By the time we get to S///, $orig is aliased to $_
The S/// non-destructively executes substitution on $_ and returns the resulting string. This is what the smartmatch will operate on
The smartmatch, following the rules for match of Str against Str, will give True or False depending on whether substitution happened (True, confusingly, meaning it didn't)

At the end of it all, we aren't getting what we actually want: the version of the string with substitution applied.

With The Given

Now that we know that S/// always works on $_ and returns the result, it's easy to come up with a whole bunch of ways that set $_ to our original string and gather back the return value of S///, but let's look at just a couple of them:

my $orig = 'meowmix';
my $new = S/me/c/ given $orig;
say $orig;
say $new;

my @orig = <meow cow sow vow>;
my @new = do for @orig { S/\w+ <?before 'ow'>/w/ };
say @orig;
say @new;

# OUTPUT:
# meowmix
# cowmix
# [meow cow sow vow]
# [wow wow wow wow]

The first one operates on a single value. We use the postfix form of the given block, which lets us avoid the curlies (you can use with in place of given with the same results). From the output, you can see the original string remained intact.

The second example operates on a whole bunch of strings from an Array and we use the do keyword to execute a regular for loop (that aliases to $_ in this case) and assign the result to the @new array. Again, the output shows the originals were not touched.

Adverbs

The S/// operator—just like s/// and some methods—lets you use regex adverbs:

given 'Lörem Ipsum Dolor Sit Amet' {
    say S:g      /m/g/;  # Löreg Ipsug Dolor Sit Aget
    say S:i      /l/b/;  # börem Ipsum Dolor Sit Amet
    say S:ii     /l/b/;  # Börem Ipsum Dolor Sit Amet
    say S:mm     /o/u/;  # Lürem Ipsum Dolor Sit Amet
    say S:nth(2) /m /g/; # Lörem Ipsug Dolor Sit Amet
    say S:x(2)   /m /g/; # Löreg Ipsug Dolor Sit Amet
    say S:ss/Ipsum Dolor/Gipsum\nColor/; # Lörem Gipsum Color Sit Amet
    say S:g:ii:nth(2) /m/g/; # Lörem Ipsug Dolor Sit Amet
}

As you can see, they are in the form of :foo that is added after the S part of the operator. You can use whitespace liberally and several adverbs can be used at the same time. Here are their meanings:

:g—(long alternative: :global) global match: replace all occurances
:i—case insentive match
:ii—(long alternative: :samecase) preserve case: regardless of the case of letter used as a substitute, the original case of the letter being replaced will be used
:mm—(long alternative: :samemark) preserve mark: in the example above, the diaeresis that was on letter o was preserved and applied to the replacement letter u
:nth(n)—replace only nth occurance
:x(n)—replace at most n occurances (mnemonic: "x as in times")
:ss—(long alternative: :samespace) preserve space type: the type of whitespace character is preserved, regardless of whitespace characters used in the replacement string. In the example above, we replaced with a new line, but the original space was kept

Method Form

Operator S/// is nice, but using it is somewhat awkward at times. Don't fear, Perl 6 provides .subst method for all your substitution needs and delightful .subst/.substr confusion. Here's what its use looks like:

say 'meowmix'.subst: 'me', 'c';
say 'meowmix'.subst: /m./, 'c';

# OUTPUT:
# cowmix
# cowmix

The method takes either a regex or a plain string as the first positional argument, which is the thing it'll look for in its invocant. The second argument is the replacement string.

You can use the adverbs as well, by simply listing them as named Bool arguments, with a slight caveat. In S/// form, adverbs :ss and :ii imply the presence of :s (make whitepsace significant) and :i (case-insensitive match) adverbs respectively. In method form, you have to apply those to the regex itself:

given 'Lorem Ipsum Dolor Sit Amet' {
    say .subst: /:i l/, 'b', :ii;
    say .subst: /:s Ipsum Dolor/, "Gipsum\nColor", :ss;
}

# OUTPUT:
# Borem Ipsum Dolor Sit Amet
# Lorem Gipsum Color Sit Amet

Method Form Captures

Captures aren't alien to substitution operations, so let's try one out with the method call form of substitution:

say 'meowmix'.subst: /me (.+)/, "c$0";

# OUTPUT:
# Use of Nil in string context  in block <unit> at test.p6 line 1
# c

Not quite what we were looking for. Our replacement string is constructed even before it reaches the .subst method and the $0 variable inside of it actually refers to whatever it is before the method call, not the capture in the .subst regex. So how do we fix this?

The second argument to .subst can also take a Callable. Inside of it, you can use the $0, $1, ... $n variables the way they were meant to and get correct values from captures:

say 'meowmix'.subst: /me (.+)/, -> { "c$0" };

# OUTPUT:
# cowmix

Here, we've used a pointy block for our Callable, but WhateverCode and subs will work too. They will be called for each substitution, with the Match object passed as the first positional argument, if you need to access it.

Conclusion

The S/// operator in Perl 6 is the brother of s/// operator that instead of modifying the original string, copies it, modifies, and returns the modified version. The way to use this operator differs from the way non-destructive substitution operator works in Perl 5. As an alternative, a method version .subst is available as well. Both method and operator form of substitution can take a number of adverbs that modify the behaviour of it, to suit your needs.

2 comments

Perl 6 Types: Made for Humans

By Zoffix Znet on April 6, 2016 2:02 AM under Perl 6, Perl 6, perl6

In my first college programming course, I was taught that Pascal language has Integer, Boolean, and String types among others. I learned the types were there because computers were stupid. While dabbling in C, I learned more about what int, char, and other vermin looked like inside the warm, buzzing metal box under my desk.

Perl 5 didn’t have types, and I felt free as a kid on a bike, rushing through the wind, going down a slope. No longer did I have to cram my mind into the narrow slits computer hardware dictated me to. I had data and I could do whatever I wanted with it, as long as I didn’t get the wrong kind of data. And when I did get it, I fell off my bike and skinned my knees.

With Perl 6, you can have the cake and eat it too. You can use types or avoid them. You can have broad types that accept many kinds of values or narrow ones. And you can enjoy the speed of types that represent the mind of the machine, or you can enjoy the precision of your own custom types that represent your mind, the types made for humans.

Gradual Typing

my       $a = 'whatever';
my Str   $b = 'strings only';
my Str:D $c = 'defined strings only';
my int   $d = 16; # native int

sub foo ($x) { $x + 2 }
sub bar (Int:D $x) returns Int { $x + 2 }

Perl 6 has gradual typing, which means you can either use types or avoid them. So why bother with them at all?

First, types restrict the range of values that can be contained in your variable, accepted by your method or sub or returned by them. This functions both as data validation and as a safety net for garbage data generated by incorrect code.

Also, you can get better performance and reduced memory usage when using native, machine-mind types, providing they’re the appropriate tool for your data.

Built-In Types

There’s a veritable smörgåsbord of built-in types in Perl 6. If the thing your subroutine does makes sense to be done only on integers, use an Int for your parameters. If negatives don’t make sense either, limit the range of values even further and use a UInt—an unsigned Int. On the other hand, if you want to handle a broader range, Numeric type may be more appropriate.

If you want to drive closer to the metal, Perl 6 also offers a range of native types that map into what you’d normally find with, say, C. Using these may offer performance improvements or lower memory usage. The available types are: int, int8, int16, int32, int64, uint, uint8, uint16, uint32, uint64, num, num32, and num64. The number in the type name signifies the available bits, with the numberless types being platform-dependent.

Sub-byte types such as int1, int2, and int4 are planned to be implemented in the future as well.

Smileys

multi foo (Int:U $x) { 'Y U NO define $x?'         }
multi foo (Int:D $x) { "The square of $x is {$x²}" }

my Int $x;
say foo $x;
$x = 42;
say foo $x;

# OUTPUT:
# Y U NO define $x?
# The square of 42 is 1764

Smileys are :U, :D, or :_ appended to the type name. The :_ is the default you get when you don’t specify a smiley. The :U specifies undefined values only, while :D specifies defined values only.

This can be useful to detect whether a method is called on the class or on the instance by having two multies with :U and :D on the invocant. And if you work at a nuclear powerplant, ensuring your rod insertion subroutine never tries to insert by an undefined amount is also a fine thing, I imagine.

Subsets: Tailor-Made Types

Built-in types are cool and all, but most of the data programmers work with doesn’t match them precisely. That’s where Perl 6 subsets come into play:

subset Prime of Int where *.is-prime;
my Prime $x = 3;
$x = 11; # works
$x = 4;  # Fails with type mismatch

Using the subset keyword, we created a type called Prime on the fly. It’s a subset of Int, so anything that’s non-Int doesn’t fit the type. We also specify an additional restriction with the where keyword; that restriction being that .is-prime method called on the given value must return a true value.

With that single line of code, we created a special type and can use it as if it were built-in! Not only can we use it to specify the type of variables, sub/method parameters and return values, but we can test arbitrary values against it with the smartmatch operator, just as we can with built-in types:

subset Prime of Int where *.is-prime;
say "It's an Int"  if 'foo' ~~ Int;   # false, it's a Str
say "It's a prime" if 31337 ~~ Prime; # true, it's a prime number

Is your “type” a one-off thing you just want to apply to a single variable? You don’t need to declare a separate subset at all! Just use the where keyword after the variable and you’re good to go:

multi is-a-prime (Int $ where *.is-prime --> 'Yup' ) {}
multi is-a-prime (Any                    --> 'Nope') {}

say is-a-prime 3;     # Yup
say is-a-prime 4;     # Nope
say is-a-prime 'foo'; # Nope

The --> in the signature above is just another way to indicate the return type, or in this case, a concrete returned value. So we have two multies with different signatures. First one takes an Int that is a prime number and the second one takes everything else. With exactly zero code in the bodies of our multies we wrote a subroutine that can tell you whether a number is prime!!

Pack it All Up for Reuse

What we’ve learned so far is pretty sweet, but sweet ain’t awesome! You may end up using some of your custom types quite frequently. Working at a company where product numbers can be at most 20 characters, following some format? Perfect! Let’s create a subtype just for that:

subset ProductNumber of Str where { .chars <= 20 and m/^ \d**3 <[-#]>/ };
my ProductNumber $num = '333-FOOBAR';

This is great, but we don’t want to repeat this subset stuff all over the place. Let’s shove it into a separate module we can use. I’ll create /opt/local/Perl6/Company/Types.pm6 because /opt/local/Perl6 is the path included in module search path for all the apps I write for this fictional company. Inside this file, we’ll have this code:

unit module Company::Types;
my package EXPORT::DEFAULT {
    subset ProductNumber of Str where { .chars <= 20 and m/^ \d**3 <[-#]>/ };
}

We name our module and let our shiny subsets be exported by default. What will our code look like now? It’ll look pretty sweet—no, wait, AWESOME—this time:

use Company::Types;
my ProductNumber $num1 = '333-FOOBAR'; # succeeds
my ProductNumber $num2 = 'meow';       # fails

And so, with a single use statement, we extended Perl 6 to provide custom-tailored types for us that match perfectly what we want our data to be like.

Awesome Error Messages for Subsets

If you’ve been actually trying out all these examples, you may have noticed a minor flaw. The error messages you get are Less Than Awesome:

Type check failed in assignment to $num2;
expected Company::Types::EXPORT::DEFAULT::ProductNumber but got Str ("meow")
in block <unit> at test.p6 line 3

When awesome is the goal, you certainly have a way to improve those messages. Pop open our Company::Types file again, and extend the where clause of our ProductNumber type to include an awesome error message:

subset ProductNumber of Str where {
    .chars <= 20 and m/^ \d**3 <[-#]>/
        or warn 'ProductNumber type expects a string at most 20 chars long'
            ~ ' with the first 4 characters in the format of \d\d\d[-|#]'
};

Now, whenever the thing doesn’t match our type, the message will be included before the Type check... message and the stack trace, providing more info on what sort of stuff was expected. You can also call fail instead of warn here, if you wish, in which case the Type check... message won’t be printed, giving you more control over the error the user of your code receives.

Conclusion

Perl 6 was made for humans to tell computers what to do, not for computers to restrict what is possible. Using types catches programming errors and does data validation, but you can abstain from using types when you don’t want to or when the type of data you get is uncertain.

You have the freedom to refine the built-in types to represent exactly the data you’re working with and you can create a module for common subsets. Importing such a module lets you write code as if those custom types were part of Perl 6 itself.

The Perl 6 technology lets you create types that are made for Humans. And it’s about time we started telling computers what to do.

UPDATE:

Perl 6 will actually evaluate your where expression when checking types even for optional parameters. This can be quite annoying, due to “uninitialized” values being compared. I wrote Subset::Helper to make it easier to create subsets that solves that issue, and it provides an easy way to add awesome error messages too.

4 comments

"Wow, Perl 6!" Talk: Slides, Recording, and Answers to Questions

By Zoffix Znet on March 31, 2016 12:15 PM under Perl 6

Last night I gave a "Wow, Perl 6!" talk at the Toronto Perl Mongers, whom I thank for letting me speak, even after I got lost for 15 minutes in the building the event was hosted at and was subsequently late.

The talk is an overview of some of the cool features Perl 6 offers. If you didn't get a chance to be at the talk or to watch it via a Google Hangout, you can still get a recording of it.

You can view the slides at http://tpm2016.zoffix.com/ and the recording of the talk is on YouTube:

Synopsis

Couch Potato:

Lazy lists and their uses

Molding Your Own:

Subsets
Custom operators
Muti-dispatch

Hyperspace: Multi-core processing at a touch of a button

Hyper operators
Hyper sequence methods
Autothreaded junctions
Promises, Supplies, and Channels

How's Your Spellin'?

Grammars: Parsing made easy

Whatever, man!:

Whatever Code
Meta operators
Model6 Object Model (very brief "teaser" overview)
MOP: Meta Object Protocol
Sets, bags, and mixes

Polyglot:

NativeCall
Inline::Perl5

Not Really Advanced Things:

Hacking on the Perl 6 Compiler

Bonus Slides:

Backtraces for failures in concurrent code
Peculiarities with Rats
Proc::Async
say is for humans put is for computers
More useful objects
Built in profiler

Answers to Questions

During the talk a couple of questions were asked and I didn't know the answer at the time. I do now:

Is there a way to have better error messages when a subset doesn't match a value given?

The code in the where can be anything you want, so you can warn or fail inside the check to get a better error message. Once caveat: the argument given to callframe might be different depending on where you're performing the check. Try adjusting it:

subset Foo of Int where {
    $_ > 10_000
        or fail "You need a number more than 10,000 on "
            ~ "line {(callframe 4).line}, but you passed $_";
};

my Foo $x = 1000;

# OUTPUT:
#  You need a number more than 10,000 on line 7, but you passed 1000
#  in block <unit> at test.p6 line 2

Can you check whether or not a value fits the subset?

Yes, just smartmatch against the type/subset:

subset Even where * %% 2;
say 3 ~~ Even;
say 42 ~~ Even

# OUTPUT:
# False
# True

Can you have an infinite `Set`?

No, it tries to actually create one. Makes sense, since a Set cares about the elements in it. Sure, it's possible to special-case some forms of sequences to figure out whether an element is part of the sequence or not, but it's probably not worth it. In a more general case, you are faced with the Halting Problem. Speaking of which, here is a gotcha with the sequence operator and the upper limit:

my @seq = 0, 2 ... * == 1001;

Here, I'm using the sequence operator to create a sequence of even numbers, and I'm limiting the upper bound by when it'd be equal to 1001. But it won't ever be equal to that. To human brain, it might seem obvious that once you're over 1001, you should stop here, but to a computer it's a Halting Problem and it'll keep trying to find the end point (so it'll never complete here).

Can you kill a running Promise?

Not possible. If you need that kind of thing, you'll have to use processes, or you'll have to build the code inside the Promise so that it exposes some kind of a "should I continue working?" flag.

Links for Learning Materials and Ecosystem

Along with http://perl6intro.com/ that I mentioned during the talk, there's also Learn X in Y Minues Perl 6 page, which I personally found very useful when just starting out with Perl 6.

The Ecosystem is at http://modules.perl6.org/ you should have panda program installed, and you can install modules from the Ecosystem by typing panda install Foo::Bar

0 comments

Perl 5 Porters Mailing List Summary: February 2nd-7th

By Sawyer X on February 8, 2016 3:55 PM under p5p-summary

Hey everyone,

Following is the p5p (Perl 5 Porters) mailing list summary for the past week. Enjoy!

2 comments

< Previous 1 2 3 4 5 6 7 Next >

About coke

Perl 6 hacker

More info »

Results matching “smartmatch”

Thank You!

Completeness Criteria

Extra Deliverables

Work Performed in May

Work by Other Core Members

Tickets Fixed

Grant Commits

Performance Rakudo Commits

Non-Performance Rakudo Commits

Perl 6 Specification Commits

Perl 6 Documentation Commits

Timing

Communications

IO Action Plan Progress

Documentation and Coverage Progress

Optimizations

IO Optimizations in 2017.04.2 Done by Other Core Members:

Tickets Resolved

Bugs Fixed

IO Bug Fixes in 2017.04.2 Done by Other Core Members:

Commits

Rakudo

Perl 6 Specification

Documentation

Golf It Down

Roast It

Make it Saucy

Mix Your Fix

Pull The Trigger

A Clean Kill

Conclusion

The Basics

Getting Smarter

Passing The Buck

Multify All The Things!

An Eventful Day

Keep 'Em Commin'

Don't Wait Up

Watch Your Mouth!

It Spreads Like Butter

Send It That Way

Conclusion

The Smartmatch

With The Given

Adverbs

Method Form

Method Form Captures

Conclusion

Gradual Typing

Built-In Types

Smileys

Subsets: Tailor-Made Types

Pack it All Up for Reuse

Awesome Error Messages for Subsets

Conclusion

UPDATE:

Synopsis

Answers to Questions

Is there a way to have better error messages when a subset doesn't match a value given?

Can you check whether or not a value fits the subset?

Can you have an infinite Set?

Can you kill a running Promise?

Links for Learning Materials and Ecosystem

About coke

Search this blog

Can you have an infinite `Set`?