I promised to get my slides up afterwards but life (i.e. my doctoral thesis) got in the way. Now with the release of Mojolicious 4.0 I thought I would take the opportunity to right a wrong and get the slides up; so here they are: http://mojolicious-introduction.herokuapp.com/!
The talk is itself a Mojolicious app, the source of which is available from on GitHub. Not only are all the code snippets shown in the talk included, not only do they all run, but they are actually what is rendered by the talk (DRY++), so what you see is what you get! Please leave any feedback and ask any questions. I may not see the responses here, so feel free to ping me elsewhere if needed.
]]>Our fearless leader Sebastian Riedel (aka sri) will no doubt post a message with all the details when it ships. In the meantime, I want to share a little story of how community interaction, even at the StackOverflow level, can lead to innovation and enhancement of major projects like Mojolicious!
]]> On with the story! StackOverflow helped me learn Perl several years ago. On occasion, to pay-it-forward, I do my best to help a (polite, courteous) questioner. I hope that I can use that platform to keep a prospective Perl user in the Perl world by suggesting good tutorials or books, suggesting good practices and yes explain some of Perl’s lovable quirks. Sometimes I even get to promote some of my favorite Perl modules and projects too.My favorites though are the rare times when the question is interesting, or better still, when a question makes a real impact. The other day someone asked a seemigly simple question: using Mojolicious’ built-in event loop (which will use EV/AnyEvent if available), how can one restart a timer event. This might be useful for pushing back a timeout for example.
I posted my naive response, in which the “restart” was really stopping the old timer and starting it again. Because I was interested, I asked sri what his opinions on my solution were. Based on his response I added a second example which the timer is kept alive, but now it has to repeatedly check if it should fire. The discussion didn’t end there, however, but lasted several days, eventually leading to implementing a new feature in Mojolicious. The benefit is not just an easy way to restart a timer, but a massive performance enhancement for Mojolicious apps under high-load with high-concurrency.
With these new optimizations, sri’s example app can maintain 10_000 concurrent websocket connection with zero CPU overhead! That’s some really awesome results to obtain when starting from a humble little StackOverflow question. But indeed it’s these cases that make me love open source software even more! Communities of people asking questions and giving answers, communities of experienced programmers and newbies too can all work together to make what’s already awesome even more awesome!
My personal favorite new feature is that sending JSON over websockets, something my Galileo CMS does regularly, will be easier than ever. To send JSON from the backend to the browser, simply use the send method as always, but with the new json key
$ws->send({ json => { foo => [qw/ bar baz bat /] } });
and when JSON is expected, you can get it simply by subscribing to the new json event
$ws->on( json => sub { my ($ws, $data) = @_; ... } );
This all makes JSON over websocket a really easy way to do complex client/server interaction and makes doing it a breeze!
I’m looking forward to Mojolicious 4.0! I hope you are too!
]]>The report is included after the break.
]]> Alien::Base Final ReportWith this report I end my grant for Alien::Base. I consider it to be a reasonable success and have hope that the project will continue further. It became, as perhaps I should have expected, a larger project than anticipated; the problems were rarely the anticipated ones.
In the end Alien::Base faced two major problems: (program/script) compile-time linking of the library and the localization of the built binary library on Mach (read Mac) platforms.
The the linking problem is solved, running use Alien::MyLibrary; in the dependent module dynamically loads the module as expected.
The Mach problem should be solved, though part of the problem is that the solution must of necessity be different for proper users and CPANtesters.
Mach binaries must hard-code the path to themselves in the binary, this requires fore-knowledge of the final install location of the library.
Steps have been taken to ensure that the install location is set correctly for common (perl Build.PL && ./Build && ./Build install) use.
CPANtesters unfortunately do not run the install phase, but rather append the blib directory into @INC.
I hope that I have fixed this problem, however I have waited months and yet have seen no Mac reports on either of my test modules Acme::Alien::DontPanic and Acme::Ford::Prefect; until these reports come in I cannot assess the ability of my recent work to compensate for this so-called “blib-scheme”.
I am happy to report that I have received much interest in the project and even some patches to the code-base. I am confident that the project will continue to progress; for my part I will continue to contribute and shepherd patches/pull-requests. Certainly the system is still in a rather basic form, for example HTTP support is still rather fragile, but the major architecture exists and will continue to improve.
I would especially like to that my ever-patient project coordinator Makoto Nozaki for his help coordinating the project, and for far longer than expected. As the project progressed beyond its expected life, my Ph.D. Thesis and Defense took more and more of my time and so I want to thank both the Perl Foundation and the community at-large for being patient with me as well.
I also want to thank all of those people who have in any way encouraged or contributed to Alien::Base:
This process has been taxing at times, but still rewarding. I hope that Alien::Base will be a useful part of the CPAN toolchain in the future as the Perl renaissance progresses!
]]>Ack is a tool for searching code and text. It works much like the unix tool grep, although it is imbued with the power of Perl. To mark the release of Ack 2.0 though I wanted to mention a few one-liners that made my life easier in this stressful time.
My thesis is written in many LaTeX files and one can probably imagine that searching those files was needed regularly. The biggest is for finding non-ascii characters. As I add content from old publications or external programs, lots of non-ascii characters can often come along for the ride. LaTeX is a very old program, well pre-dating unicode, and it has a very different way of adding special characters with its own markup. In fact parts of the compiling toolchain croak with unicode characters. So I found myself using
ack '[^[:ascii:]]'
regularly. Also I had to keep a list of all the abbreviations that I had used in the paper, but of course you forget if you have them all. I used this little bash-ack conglomeration to find all sequences of two or more upper-case characters, which is how I write my abbreviations,
ack -ho '\p{Upper}{2,}' | sort | uniq
I’m sure I used many other little ackings here and there, but these were the two I could remember off-hand. Thanks to Andy and everyone who has contributed to ack!
Now go use it to make your life easier!
Visit the new site: beyondgrep.com
]]>Today I used my favorite of my modules in order to accomplish a difficult task, and in doing so I found a little bug, which I have just fixed. Which one is it? Let me introduce you to Tie::Array::CSV.
]]> Tie::Array::CSV lets you treat a CSV (or any other deliminated file that Text::CSV can handle) as a Perl array of arrayrefs. It lets you read and write just as you would any other AoA, but in the background you are CRUD-ing a csv file.Today I needed to write columns to a CSV file, in order to pass them to Gnuplot. Normally I would use PDL for this, but the columns are of uneven length, and while I’m sure there is some way to do this, I don’t know what it is. The problem of course is that if column three is longer than column two, you will need to put an empty entry in column two so that they line up correctly. Luckily I have Tie::Array::CSV.
For this example I make a little function that generates a list of random data of random length. I want to write this data as the next column in my CSV file. Lets say I want to do this 10 times making 10 columns. Here it is:
#!/usr/bin/env perl
use strict;
use warnings;
use Tie::Array::CSV;
sub get_column {
my $last = int rand 10;
return map { rand 100 } (0..$last);
}
tie my @out, 'Tie::Array::CSV', 'outfile.csv';
for my $column (0..9) {
my $row = 0;
$out[$row++][$column] = $_ for get_column();
}
See? Piece of cake! Notice the “ragged” bottom few lines, that’s the hard part, but I didn’t have to think about it at all!
I hope if you need to work with CSV data, you’ll think about using Tie::Array::CSV. Its probably not the fastest, but I think its the easiest and most self-explanatory. Oh and if the tie keyword scares you, there is also a new constructor.
Now what is your favorite module? Is it the most clever one, the most useful? Let me know!
]]>The PDL development team is pleased to announce the official release of PDL-2.006 and an updated draft of the PDL Book to accompany its release.
Of specific note:
PDL VERSION numbers now use single decimal format. This will be the standard going forward.
PDL now has three graphics options that build on all supported PDL platforms (thanks to work by Craig DeForest and David Mertens and a host of others):
PDL::Graphics::Gnuplot using the gnuplot program
PDL::Graphics::Prima based on the Prima toolkit
and
ASPerl build issues have been resolved thanks to relentless testing, verification and fixes by Rob/sisyphus and other win32 PDL users.
As always, go to http://pdl.perl.org for information about all things PDL and how to get PDL for your platform.
Enjoy!
Chris Marshall
PDL-2.006 Release Manager
]]>Yesterday I got an email from them listing a number of failures from Galileo, my CMS. I had recently pushed some bugfix releases, but it had some new, and as yet unused code and tests for that code in it. The tests passed on all my Linux systems, so I wasn’t worried about the release. Yet the failures came in. Some on Linux, some on other platforms, but not all the tests were failures and I couldn’t figure out a pattern. CPANtesters put me on to a problem but for this I needed faster results.
I had heard about Travis-CI, a free continuous integration platform based around GitHub. I set up travis testing for Galileo and sure enough it failed there. Though it was frustrating I now had failing tests that I could run at will! After much trial and error, I found that I had an undeclared dependency, but due to the way I was testing, it was throwing a seemingly unrelated error. My problem was that all my systems have the module installed and so I didn’t get the failure on my box, its a common module File::Next (used by Ack) and so many of the CPANtesters had it as well.
CPANtesters alerted me to the problem and Travis-CI let me continuously test on fresh platforms (5.10/12/14/16) until I found the problem. I love open source.
I have released Galileo 0.026 which fixes the problem. There are exciting additions to Galileo in the works, slowed only by my upcoming Ph.D. defense (which obviously takes much of my time). I hope that by this summer Galileo will have several of the most requested features you have told me that you would like.
Happy Perl-ing and remember to thank those projects and developers who make your lives easier, both in person and in public. Thanks guys!
]]>$ cpanm Galileo
$ galileo setup
$ galileo daemon
This release makes installing even better, because now when you run galileo setup you get a web interface to configure your CMS and then install the database!
Ok that image is just slightly ahead of what is now on CPAN, but its close :-)
]]>I haven’t decided, but I might try to “self-host” the talk, writing it as a Mojolcious app! To do that I had to resurrect one of my earliest CPAN releases Mojolicious::Plugin::PPI. This module does just what the name should imply, providing syntax highlighting via PPI and PPI::HTML in a handy Mojolicious plugin.
Whether or not I decide to use this for my talk, it still is handy to have around. Here is a cute example, a simple “quine” which you can run:
#!/usr/bin/env perl
use Mojolicious::Lite;
plugin 'PPI' => { toggle_button => 1 };
get '/' => sub {
my $self = shift;
$self->stash( file => __FILE__ );
$self->render('quine');
};
app->start;
__DATA__
@@ quine.html.ep
% title 'A Mojolicious "Quine"';
<!DOCTYPE html>
<html>
<head>
<title><%= title %></title>
%= javascript 'ppi.js'
%= stylesheet 'ppi.css'
</head>
<body>
<h2><%= title %></h2>
%= ppi $file
</body>
</html>
A quine is a program that prints its source code as its output. In truth, this “quine” is really a cheater, a true quine doesn’t read itself. Still, the above example renders the source code nicely highlighted by PPI (and with some added goodies like toggleable line numbers).
Just for fun, some other Perl (cheater) quines are:
@ARGV = $0; print for <>
and
seek DATA,0,0; print for <DATA>
__DATA__
But those outputs aren’t syntax highlighted. For a real one see the comments.
]]>I just wanted to drop a quick note to announce #galileo on irc.perl.org as a place to talk about Galileo my CPAN installable CMS and its new companion project GalileoSend which isn’t quite to CPAN yet, but it will be soon.
GalileoSend will make sending files via websocket easy! So far the package includes a command-line sender (client) and receiver (listener/server), a Mojolicious plugin and a javascript client. Futher GalileoSend is a protocol spec, which means if you want to write a client or server in your other favorite language, or a plugin for your favorite framework, its very possible.
Interested? Drop on by and chat about it!
]]>The protocol itself is language independent for both the client and server side, assuming that both can open a websocket connection and send JSON (as text) and binary data over it. Since communication by websocket is cheap, 2-way communication is highly encouraged throughout the transfer and positive confirmation of receipt is required.
Further, I have written a javascript client-side implementation (which could be used for any server) and a non-blocking Mojolicious server-side implemenation (which could be used for any client).
Read on (examples!) …
]]> See what they look like together:Events are fired on both side for most of the important occurances. And the end result is nice and clean.
It still needs some polish. The Mojolicious plugin still needs tests and documentation, but I expect this to hit CPAN soon.
Finally I hope that the GalileoSend spec will be useful. I have found other websocket uploaders but most depend on the server implementation (usually node.js, example). I wrote it with that hope in mind.
Please comment, contribute, use etc!
]]>chromatic petdance and I’m sure others have suggested that rather than infight (which I don’t believe I’m doing, btw) is to make something great and show it off. While I don’t think this is enough to raise Perl’s profile ourside of our community, I have seen and IMHO done some very cool things this week:
Things I’ve Seen
Things I’ve Done
It may not be enough, but its always nice to share what you and others are doing. Go Perl!
]]>Note: I’m now not talking about soon, or what it would be named/numbered. I’m talking about ever.
Let me tell you my story, some of you may know it. I’m a Ph.D. candidate in Physics. Programming is woefully ignored in science education now. All my professors learned FORTRAN in courses during their Ph.D. (or B.S. in some cases) but now, given the ease of Mathematica, we seem to be expected to pick it up as we go on.
One day I had a nutty idea, I wanted to parse one of the temporary files created during LaTeX compilation. A friend, who knew next to nothing about Perl, suggested I needed Perl and Regexes. I can here you out there, saying “now I had two problems”. Not so. I learned, I improved and finally my first Perl script was out in the world. It doesn’t bear much resemblance to its current form but it worked.
This project had nothing to do with science, but I was awed by the power I now possessed. I went to start using Perl in my work. Perl finally solved a near-intractable programming problem in Mathematica with relative (hmmmm) ease. Physics::UEMColumn now forms the basis of my Ph.D. thesis which I am defending this spring.
To get that simulation working I needed more tools, so I wrote them, and those tools needed tools so I wrote them and suddenly I discovered that I had something I could give back, so I’m writing it too. Further I often need PDL, and amazing numerical package for Perl. It really needs some internal TLC, which David Mertens, project pumpking Chris Marshall and others are planning.
Why am I telling you this? Because in 2009 I became a Perl programmer, and I fell for language hard. I love Perl. I came for regexes, but I stayed because it works well with the way I think. I use it for lots of things now. Unfortunately few others have come with me.
Chromatic implores us to write good software and show it off. For these years I have tried, and it doesn’t work. I have given talks at my university, I have told friends, I have used it to help coworkers, I share with the community. I have recruited one person. I have converted none. Most people who know Python or ROOT (C++) laugh at me. Its not enough.
Until yesterday, I had a steely resolve. Even in the face of Stevan’s recent pronouncement and embarking on replacing the Perl internals with Scala, which would probably kill or at least maim most of my work. Moe wasn’t the future of Perl, just an offshoot, right?
Yesterday, for the first time, I saw a flaw. There are people out there who really think that Perl will never see another major point release. I don’t mean soon, I don’t mean for cheap marketing, I mean ever. I mean for at some point breaking some small amount of backward compatibility. For some the reason is that compatibility is too important (the future is our past), for some its that Perl 6 is the future of Perl 5 (whether spoken/believed or not) and some its just because there is no number available. All that got me to thinking …
Is the work on Perl 5 really just a shim to support CPAN? Is it really going to be an ever-growing feature pragma? Will there always need to be a magic incantation to activate some of the best Unicode support in the programming world? Will new Perlers need to be told to use good practices like strict and warnings forever? Is the herculean work of Nick Clark and Dave Mitchell just going to go for naught after a few years of a usable Perl 6 and Moe?
If Perl can never have a major point release. I think all of the above is fated to be true. I’m not asking about soon; I’m asking about ever.
In my mind I’m planning future work on the Perl we now call 5. I want to help improve the giant XS extension called PDL, possibly using another one called Prima. I intend to keep working on Alien::Base. But is it all just going to vaporize in a few years do to lack of external interest, or after being replaced by something? Why should I?
In the upcoming months, I will be looking for my first real job. It might be a post-doc appointment with Pacific Northwest National Lab (PNNL) doing the hardware/software integration for a brand new ultrafast electron microscope. Should I do it in Perl? I will only be there a year or two, is it fair to saddle them with that? Will they find someone else who can maintain it?
If I don’t do that, I might form a company, one that aids research groups in writing the software that they often cobble together, for analysis, data warehousing and search, for hardware integration. Would I do that in Perl?
If the powers that be don’t see a major point release at some point in Perl’s future, then it has none. The userbase will keep sliding away. Not purely for a number, or marketing, or perception, but because you will need too much knowledge to start using it. You will need to know the incantations to do things right, ones that we already know, but new users can’t find. How many people find a CPAN module that does just what they want, and so set out to write their Perl first script, only to fail and walk away? We can’t ever know.
I ask you p5p (whom I admire greatly) and I ask you Larry, creator, benevolent dictator, does the language we call Perl 5 have a future? At some time, will we see a major release? If not, then I don’t know what I’m working towards. Should I tie my future to this language?
I just got back from FOSDEM and heard, again, for the umpteenth time, that since Perl had 4 “major” releases (1,2,3,4) in its first few years and hasn’t had a major release since Perl 5 about 20 years ago, it’s clearly “dead”. I hear this every time I go to a conference that’s not dedicated to Perl.
…
Meanwhile, in the minds of, of, virtually every developer on the planet who’s not in our echo chamber, Perl 6 is, logically, the successor to Perl 5. Since even Duke Nukem Forever managed to get released prior to Perl 6, Perl 6 just isn’t being taken too seriously be many devs. In other words, they think Perl 5 is dying and its successor is DOA.
I agree with him. As I commented:
I think Perl 5 has had to live in the shadow of Perl 6 for too long. Now that its decided that they will be two different languages, I think its only fair to allow Perl to get the cache’ one gets from a major version increment.
This can end the “Perl is dead” chatter (outside the community anyway). After all why would you want to use Perl 5 when Perl 6 is going to replace it (says Joe Pythonista or Jane Rubyist).
Perl 7 is here! Its Plack and Catalyst/Dancer/Mojolicious(+Mango (what’s Mango?)), its Moo(se)? and DBIx::Class. Its CPANtesters and MetaCPAN and cpanm. There is so much the outside world has missed because they are waiting for Perl 6, which may come at some point but it still wont replace Perl 5. Lets show them its safe to come back and that there are LOTS of goodies to find when they do!
That said I want to take it even a little further. Perl has been proud of its backward compatibility, and well it should be! Code from many years ago runs well on very recent Perls. But what can be a blessing can also be a curse.
Perl 5 still supports many conventions and oddities from Perl 4 and before. There are some silly things like ' as a package separator, which most people neither use nor trip over. Then again there are some real things, like indirect objects notation, default script/handle encoding, new keywords.
Olof Stockhaus made a compelling case that Perl 5 might want to use the jump to Perl 7 to make some breaking changes. For example, among other things he suggests waiting for the p5mop project. However, I think I agree with Damien Krotkine in saying that that sounds like a great target for Perl 8 which could come soon afterwards. There are many things we wish for a future Perl, but until they are ready, there still could be some changes. I think Perl 7 could be a great oportunity to do minor, but important, breaking changes. Then let Perl 8 come in later with a MOP or a proper language spec. Either way, without a major version increase its hard to make breaking changes. Without a major version to increase, we have been stuck.
Now, some have suggested that we should drop the 5 from the version number and add it to the name. Therefore we would have Perl5 version 20.0 and Perl6 version 0.1. The problem is that this still doesn’t make the case that Perl6 won’t supplant Perl5. Those people like to make the case that Java dropped the 1 so that Java 1.6 became Java 6. However this comes back to my first point. They realized that 1.5 and 1.6 (or whatever was the actual change) was going to have major changes and should be a major release. Why they didn’t simply call it Java 2 I’m not sure. What I do know is that they weren’t trying to convince people to use Java1 version 6 when something called Java2 existed.
To comment on using the year as the version number I think there are two problems with that. First, it doesn’t really signify that Perl6 is not the successor. Secondly, what it does seem to signify is that the Perl community is going to abandon the stability that is one of its strongest suits. Yes I’m advocating some breaking changes above, but rather small ones, things that will help more than they hurt to fix. I don’t think we want to imply that there will be breaking changes once per year, we can still use feature to scope changes to Perl 7 through sub-major version releases. That said, breaking changes once a decade is probably ok.
Perhaps this is the root problem. The semantics of versioning are an important tool to inform the outside world what the state of Perl is!
Anyway here’s what I propose. Say we do implement the above, so now we have two languages Perl5 and Perl6. However, since Perl5 version 20 is mostly compatible with Perl version 5 perhaps it could be allowed to keep the name. Then it seems odd that Perl should jump versions from 5 to 20, so why not just move to the next available number 7. So now we have Perl 7 successor to Perl 5 and Perl6 a new language.
This does not signify the death of Perl 6 but the life of Perl. It just so happened that Perl 6 became its own language and started its own branch in the Perl family tree. Now that we know that the success of Perl 5 and Perl 6 no longer depend on each other, why should their version numbers?
After the break, my initial list of realistic goals for Perl 7
]]> To be worthy of a breaking release, here are some things that I would want to see for a Perl 7 release. Note that I’m assuming that most of these things are already implemented and that new big niceties would be targeted for Perl 8.use utf8::all, and the unicode_ features)feature enabled by default' as a package separatorThings I would like for Perl 7
no indirect by default). This would likely mean special casing (s)?print(f)?return @array and return ( <list> ) behave the same in scalar context (whichever way) (maybe this is too advanced, and thus should be for Perl 8)Do you have any others? I’m sure there are more to come!
]]>In this report I want to focus first on the high points. Far and away my high point has been that in my apparent absence others have taken up the mantle. I have gotten bug reports and pull requests from preaction, giatorta, amannb, mokko, tobyink and as always productive conversations with David Mertens. It helps in a tough project like this has been to see the excitement of other developers, waiting for me to finish this project.
Several bugs have been fixed, others have been identified. I hope to have another dev release out soon. Further I plan to release a version of Alien::GSL which depends on Alien::Base to CPAN not long after that.
Now for the bad news. While it seems that Alien::Base is coming to a preliminary form of “completion” I have started to notice a disturbing trend coming from CPANtesters. Let me preface this by saying I think CPANtesters might be the killer app of CPAN; it is spectacular and I have said so before. There is a problem though.
Because I wanted to be sure that testing was done with properly installed modules, I created two phony distributions: Acme::Alien::DontPanic which provides a tiny C library of my own creation, and Acme::Ford::Prefect which relies on it. For this reason, when considering the test results for Alien::Base, the results of these modules must be considered as well.
As I have noted before, certain platforms (Mac OSX) need to know the full path of the library’s final install location at build time. The only way I was able to overcome this problem was to have Alien::Base build the library in a temporary location and reference the final location in the build system. Then when the module is installed, the build tool’s install command is issued. The library’s build system therefore avoids the Perl build system. Let me say this a different way, the built library never lands in a blib directory before moving to its final location.
Now I want you to note, that I have taken special care to make sure that Alien::Base refreshes the installation packlist and all-in-all there should have been no adverse effects from this plan. It wasn’t until I started to get failing tests for the packlist handling on Acme::Ford::Prefect that I noticed the problem. While I was able to prevent those problems from being reported, they belied a deeper problem.
This brings us back to CPANtesters. When testing some module, many of the CPANtesters smoke testers don’t actually install the dependent modules. What they do is build each dependent module, then add all the relevant blib directories to the @INC array. This is a clever system, it means that a fresh dependency chain is available for each test, it means that the testing platform is not affected by installing all these modules. Unfortunately its not testing a real install environment. For most people and for most modules this is fine. For Alien::Base as currently constituted however, its a very real distinction.
On platforms where this library path information is not problematic, Alien::Base will work correctly, even under CPANtesters blib scheme. For platforms that suffer from this problem, I believe that Alien::Base will work correctly, but not under the blib scheme. Therefore no dependent modules will be able to successfully dynamically link to the provided C library.
Until Alien::Base-based modules can install and test on the three major platforms, I hesitate to call Alien::Base completed. I think I’m rather close to declaring victory on 5 of those 6 goals, however if CPANtesters testing on Mac is broken, I’m still stuck saying that there is work to be done before any victory may be declared.
Perhaps there is some way that CPAN testers can work with me to avoid this problem, admittedly I haven’t talked to them about this. If anyone with more knowledge on the subject wants to comment below, email me or file a github issue, please do.
]]>