PetaMem

pperl 0.6.23 is released. As announced in the addendum to the last release post, the complete record of what changed is no longer this post's job: it is in the changelog, one page per version, at https://perl.petamem.com/docs/eng/changelog/index.html. That leaves the post free for three things. One is a rant, about licences and how pperl came to have its own line editor. One is a prompt: pperl has an interactive mode now, a REPL for Perl that keeps your variables from line to line and tells you whether your loop was compiled. And one is speed: MIME::Base64, Digest::SHA and Encode, three modules whose perl5 versions are C, now run significantly faster in pperl than in perl with that C.

RANT

I have been on Linux since 1994, and pperl runs on Linux and nothing else. When Steve Ballmer said in 2001 that "Linux is a cancer that attaches itself in an intellectual property sense to everything it touches", I was on the barricades with everybody else. He had the wrong addressee. What he described was not Linux, it was the viral nature of the GPL. And in retrospect - I hate to admit it - he was right.

GPL-3 and the AGPL are cancer. I do not say this because I dislike the FSF. I say it because I do like it, and would wish it did not work towards its own demise.

A short reminder of what they are. The GPL-2 of 1991 says: whoever distributes a work based on GPL code distributes all of it under the GPL, with source. Its text does not make a program that links to a GPL library at run time such a work; the FSF claimed that, the licence did not say it. The GPL-3 of 2007 writes it into the licence: the source it demands includes the dynamically linked parts a work is designed to require. It also added rules about patents, about DRM, and about the devices the software runs on, which must let the user install a modified version. The AGPL-3 extends the obligation from distributing a program to letting people use it over a network: run a modified version as a service and its source is owed to every user of the service. Linux itself, the very thing Ballmer was talking about, never moved to GPL-3; it is GPL-2 only to this day. And a good part of the industry has a policy of not touching AGPL code at all.

When I look at the talks and speeches of Richard Stallman over the past years, I have to conclude that the man is not interested in software development. No algorithms, no software engineering, no "how do we do X to move field Y forward". It is all about licences, period.

The hubris of the GPL-3 might have worked for a component that is indispensable, where whoever builds on it concludes: without the shoulders of this giant we cannot do it, so be it, we accept the terms. GNU Readline is the textbook case. In 1992 CLISP linked against it. The licence text did not settle whether that made CLISP a work based on readline; the FSF said it did, and CLISP's authors, no more interested in a dispute than we are, gave in and put CLISP under the GPL. The FSF has been citing that story ever since, as the argument for putting libraries under the GPL instead of the LGPL.

That does not work any more.

Readline is GPL-3 since version 6.0. In the FSF's reading, pperl loading libreadline at run time, even dynamically, would put pperl under the GPL-3. We disagree. But we are more interested in software development than in teaching the FSF legal norms, so this release contains a clean-room reimplementation of readline in Rust: written from the readline manual, with nothing taken from the GPL source. About 15,000 lines, built in a couple of days, and checked against the real thing: 5,866 test cases run under perl 5.44 with GNU Readline 8.3 and under pperl with ours, compared line, cursor and screen after every key, and they agree on all of them. pperl does not need libreadline, libhistory or a curses library. Goodbye.

In times when one can do such a thing, the permissive licences - the Artistic License, MIT, BSD and their relatives - become the better licences, because they make people actually use your software, under any condition (so it stays relevant). Perl has been under "the Artistic License or the GPL, your choice" since 1991 and therefore Perl was doing it right all along. The FSF seems to have chosen a path that may yet turn out to be fatal.

REPL

Python has one, node has one, Ruby has irb. Perl has perl -de 0, which is the debugger standing in for a prompt: every line is evaluated in its own scope, so the my $x of one line is gone by the next. CPAN has done better for years - Devel::REPL, Reply, PDL's perldl - but each of them is something to install first, on a machine that has a perl and a compiler for their dependencies.

pperl 0.6.23 has the prompt built in:

pperl --shell

or, under its own name, after one symlink:

ln -s "$(command -v pperl)" ~/bin/ipps
ipps

A bare pperl is unchanged and reads a program from standard input, as perl does. The name is short for "interactive PetaPerl shell", and it is the fourth candidate: psh is an old CPAN distribution, pps belongs to OpenPMIx, and ppsh turned out to be a program of a Haskell pretty-printer, which we found out when it ran in place of ours.

The one idea

A line at the prompt is Perl, compiled and run as it would be in a file, and what it leaves behind stays: my variables, subs, loaded modules, pragmas, the current package.

ipps> my @primes = (2, 3, 5, 7)
(2, 3, 5, 7)
ipps> sub square { $_[0] ** 2 }
ipps> map { square($_) } @primes
(4, 9, 25, 49)

Values are shown as you would write them in Perl. A number is bare and a string is quoted, so 12 and '12' are told apart, and the quotes are the ones that need the fewest escapes. The last value shown is in $ans, or @ans when there were several.

ipps> { name => "ada", langs => ["perl", "rust"] }
{ langs => ['perl', 'rust'], name => 'ada' }
ipps> "it's"
q{it's}

Unfinished input gets a second prompt, errors are printed and the session goes on, and Tab completes from what the session holds: a variable, a key of one of its hashes, a method of an object, a module after use, a file name inside a string. Completion runs nothing; it reads what exists.

Asking the JIT

One command exists only because the runtime has a compiler. :jit runs a line and reports what the JIT did meanwhile:

ipps> :jit my $t = 0; for my $i (1 .. 1_000_000) { $t += $i * 2 } $t
1000001000000
jit: 1 loop compiled, 1 new loop seen (1 of 70 since the start)

Whether a given loop of yours gets compiled used to be a question for --jit-stats on a whole program. At the prompt it is one line, and the next line can be the loop rewritten. The report is deliberately modest: it says that a loop was compiled, not why another one was not. :time does the same for wall-clock and CPU time.

The other commands start with a colon as well: :vars lists the session's variables, :dump shows a value uncut, :source the code of a sub with the pragmas it was compiled under, :which where a module comes from, :save writes the session out as a script, and :install fetches a module from CPAN with pperl's own client. A line that starts with ! and a space goes to your shell. A start file, ~/.pperl/ipps/rc.pl, is ordinary Perl, and it can define macros and commands of your own.

Where a line is not a file

A few things in Perl last exactly as long as the code around them, and at the prompt that is one line. $1 belongs to the line that matched, local at the top of a line ends with it, a my sub is gone by the next prompt. The tutorial has a chapter on these, so that they are learned there and not by surprise.

What it stands on

ipps is written in Perl. The part that evaluates a line and keeps its lexicals is a module of its own, Peta::Shell::Session, built into the binary. Under it is Peta::ReadLine with the line editor from the rant above: GNU Readline 8.3's commands, key bindings, init file, history and completion, moving by characters in a UTF-8 locale. Since nothing is loaded from the system, line editing works on a machine without a readline and in the static build. Term::ReadLine::Gnu is an alias for it, so the perl debugger and CPAN.pm get the same editor.

The tutorial, "ipps, the interactive mode", is in the documentation in all 41 languages: https://perl.petamem.com/docs/eng/tutorial/ipps/index.html.

This is the first of three layers. The second is a shell layer: programs on the PATH callable as subs, cd, globbing. The third is pipes that carry data structures instead of text. No dates.

SPEED

Speed is the project's thesis, and the first of the three products that are sold together under the name XS. Parity with the XS was never the goal for pperl's native modules; this release has three that leave it behind.

All figures are throughput in MB/s of input on a Xeon E-2276M, six cores, measured at Perl level: a Perl script calling the module, under perl 5.44 with the XS module and under pperl, every output identical between the two. Wall time, the best of five runs, not pinned, with other work on the machine. Ratios are pperl/perl, so above 1 pperl is faster. "Default" is pperl as started, which here means six threads; "1 thread" holds it to one.

MIME::Base64, 256 MiB:

encode_base64 and decode_base64 work 12 or 24 bytes at a time with the processor's SSSE3 or AVX2 instructions. The bit arithmetic is that of the base64-simd crate, after Mula and Lemire; the handling of MIME's 76-column lines is ours, and a large input is encoded and decoded on several threads. Outside Perl, on a Pentium J4205, the kernel by itself is ahead of base64-simd on all five measurements we take. Every code path is checked against Base64.xs: every length, every byte value at every position.

Digest::SHA, 256 MiB, functional interface:

SHA.xs computes in portable C. pperl uses the processor's SHA instructions where it has them, and on a processor without them, such as this Xeon, transforms of its own that compute the message schedule four words at a time in vector registers. A hash is one sequential stream, so the threads have nothing to do here; the two pperl columns differ by noise. This is the smallest lead of the three.

Encode, single-byte encodings:

encode and decode for iso-8859-x, the cp125x code pages, koi8 and the other single-byte encodings run on two flat tables, with a copy for runs of ASCII and a large conversion split over the cores. The first two rows are the honest measure of encode: a string held as UTF-8, which is what text read through an :encoding layer or built from wide characters is, and every character converted, twice as fast on one thread and 5.5 to 8.6 times with the threads. The byte-string rows below them are not the same work. perl converts such a string to UTF-8 before encoding it; pperl sees that its latin1 is the string itself and copies it, and the same holds for pure ASCII. 14 to 68 times is a copy against a conversion, and should not be quoted as "Encode is 45 times faster". Whatever has no entry goes the way Encode.xs goes, so the results, the check modes and the error messages are perl's. The test covers all 103 table-driven encodings with all nine check modes.

Across the 23 rows of our benchmarks for the digest and encoding modules, 1 KiB to 8 MiB in operations per second, the geometric mean is 3.9 times perl on one thread and 5.0 times with the threads. No row is below perl. The nearest to parity are Base64 on short strings, 1.03 to 1.1 times, and Digest::MD5, 1.04 to 1.05 times.

zlib, one more: Compress::Raw::Zlib is built on zlib-rs, a Rust implementation with zlib's interface, where perl builds its own copy of zlib's C. Against zlib 1.3.2, compression is 1.7 to 5.7 times as fast and decompression 1.7 to 2.3 times. The price is stated in the changelog: the compressed bytes are valid for every zlib, but not the bytes perl writes.

None of this is XS, and apart from zlib's bytes, none of it changes a result.

Statistics

peta-perl 0.6.23 represents approximately 17 days of development since 0.6.20 and contains approximately 5,347,000 lines of changes across 32,000 files from 1 author.

Excluding auto-generated files, documentation, assets, vendored modules and tooling, there were approximately 337,000 lines of changes to 3,800 .rs, .pl, .pm and .t files.

Get 0.6.23 here: https://perl.petamem.com/

• Richard C. Jelinek, PetaMem s.r.o.

PS: This blog entry is under the AGPL, all your blogs and server code are belong to us. ;-)