As a newer member of the Mojolicious Core Development Team, I am more than usually excited for a Mojolicious release. This is because the next major release, version 4.0, is set to ship very soon! For those of you who don’t know, Mojolicious is a modern Perl web framework which is lightweight and easy to get started learning and using, while containing features that are cutting-edge. It’s asynchronous/non-blocking to the core, websockets work out of the box, comes with built-in DOM/JSON/UserAgent, etc etc.

Our fearless leader Sebastian Riedel (aka sri) will no doubt post a message with all the details when it ships. In the meantime, I want to share a little story of how community interaction, even at the StackOverflow level, can lead to innovation and enhancement of major projects like Mojolicious!

What is the favorite module you have released to CPAN? For me, its not some shiny CMS or fancy scientific simulation. In fact, mine is probably horribly inefficient, maybe even a little evil, but I like this one best because it is clever.

Today I used my favorite of my modules in order to accomplish a difficult task, and in doing so I found a little bug, which I have just fixed. Which one is it? Let me introduce you to Tie::Array::CSV.

Today I’m happy to make public the work I’ve been doing to make some kind of “standard” for sending files over websockets. I call it GalileoSend because it was created for the Galileo CMS.

The protocol itself is language independent for both the client and server side, assuming that both can open a websocket connection and send JSON (as text) and binary data over it. Since communication by websocket is cheap, 2-way communication is highly encouraged throughout the transfer and positive confirmation of receipt is required.

Further, I have written a javascript client-side implementation (which could be used for any server) and a non-blocking Mojolicious server-side implemenation (which could be used for any client).

Read on (examples!) …

TL;DR: Not if it can never have a major release. Please read on.

Note: I’m now not talking about soon, or what it would be named/numbered. I’m talking about ever.

Let me tell you my story, some of you may know it. I’m a Ph.D. candidate in Physics. Programming is woefully ignored in science education now. All my professors learned FORTRAN in courses during their Ph.D. (or B.S. in some cases) but now, given the ease of Mathematica, we seem to be expected to pick it up as we go on.

One day I had a nutty idea, I wanted to parse one of the temporary files created during LaTeX compilation. A friend, who knew next to nothing about Perl, suggested I needed Perl and Regexes. I can here you out there, saying “now I had two problems”. Not so. I learned, I improved and finally my first Perl script was out in the world. It doesn’t bear much resemblance to its current form but it worked.

This project had nothing to do with science, but I was awed by the power I now possessed. I went to start using Perl in my work. Perl finally solved a near-intractable programming problem in Mathematica with relative (hmmmm) ease. Physics::UEMColumn now forms the basis of my Ph.D. thesis which I am defending this spring.

To get that simulation working I needed more tools, so I wrote them, and those tools needed tools so I wrote them and suddenly I discovered that I had something I could give back, so I’m writing it too. Further I often need PDL, and amazing numerical package for Perl. It really needs some internal TLC, which David Mertens, project pumpking Chris Marshall and others are planning.

Why am I telling you this? Because in 2009 I became a Perl programmer, and I fell for language hard. I love Perl. I came for regexes, but I stayed because it works well with the way I think. I use it for lots of things now. Unfortunately few others have come with me.

Chromatic implores us to write good software and show it off. For these years I have tried, and it doesn’t work. I have given talks at my university, I have told friends, I have used it to help coworkers, I share with the community. I have recruited one person. I have converted none. Most people who know Python or ROOT (C++) laugh at me. Its not enough.

Until yesterday, I had a steely resolve. Even in the face of Stevan’s recent pronouncement and embarking on replacing the Perl internals with Scala, which would probably kill or at least maim most of my work. Moe wasn’t the future of Perl, just an offshoot, right?

Yesterday, for the first time, I saw a flaw. There are people out there who really think that Perl will never see another major point release. I don’t mean soon, I don’t mean for cheap marketing, I mean ever. I mean for at some point breaking some small amount of backward compatibility. For some the reason is that compatibility is too important (the future is our past), for some its that Perl 6 is the future of Perl 5 (whether spoken/believed or not) and some its just because there is no number available. All that got me to thinking …

Is the work on Perl 5 really just a shim to support CPAN? Is it really going to be an ever-growing feature pragma? Will there always need to be a magic incantation to activate some of the best Unicode support in the programming world? Will new Perlers need to be told to use good practices like strict and warnings forever? Is the herculean work of Nick Clark and Dave Mitchell just going to go for naught after a few years of a usable Perl 6 and Moe?

If Perl can never have a major point release. I think all of the above is fated to be true. I’m not asking about soon; I’m asking about ever.

In my mind I’m planning future work on the Perl we now call 5. I want to help improve the giant XS extension called PDL, possibly using another one called Prima. I intend to keep working on Alien::Base. But is it all just going to vaporize in a few years do to lack of external interest, or after being replaced by something? Why should I?

In the upcoming months, I will be looking for my first real job. It might be a post-doc appointment with Pacific Northwest National Lab (PNNL) doing the hardware/software integration for a brand new ultrafast electron microscope. Should I do it in Perl? I will only be there a year or two, is it fair to saddle them with that? Will they find someone else who can maintain it?

If I don’t do that, I might form a company, one that aids research groups in writing the software that they often cobble together, for analysis, data warehousing and search, for hardware integration. Would I do that in Perl?

If the powers that be don’t see a major point release at some point in Perl’s future, then it has none. The userbase will keep sliding away. Not purely for a number, or marketing, or perception, but because you will need too much knowledge to start using it. You will need to know the incantations to do things right, ones that we already know, but new users can’t find. How many people find a CPAN module that does just what they want, and so set out to write their Perl first script, only to fail and walk away? We can’t ever know.

I ask you p5p (whom I admire greatly) and I ask you Larry, creator, benevolent dictator, does the language we call Perl 5 have a future? At some time, will we see a major release? If not, then I don’t know what I’m working towards. Should I tie my future to this language?

Recently Ovid broached the topic of allowing a future release of Perl 5 with a version number greater than 6; this is not a new argument. He further explicated the problems:

I just got back from FOSDEM and heard, again, for the umpteenth time, that since Perl had 4 “major” releases (1,2,3,4) in its first few years and hasn’t had a major release since Perl 5 about 20 years ago, it’s clearly “dead”. I hear this every time I go to a conference that’s not dedicated to Perl.

…

Meanwhile, in the minds of, of, virtually every developer on the planet who’s not in our echo chamber, Perl 6 is, logically, the successor to Perl 5. Since even Duke Nukem Forever managed to get released prior to Perl 6, Perl 6 just isn’t being taken too seriously be many devs. In other words, they think Perl 5 is dying and its successor is DOA.

I agree with him. As I commented:

I think Perl 5 has had to live in the shadow of Perl 6 for too long. Now that its decided that they will be two different languages, I think its only fair to allow Perl to get the cache’ one gets from a major version increment.

This can end the “Perl is dead” chatter (outside the community anyway). After all why would you want to use Perl 5 when Perl 6 is going to replace it (says Joe Pythonista or Jane Rubyist).

Perl 7 is here! Its Plack and Catalyst/Dancer/Mojolicious(+Mango (what’s Mango?)), its Moo(se)? and DBIx::Class. Its CPANtesters and MetaCPAN and cpanm. There is so much the outside world has missed because they are waiting for Perl 6, which may come at some point but it still wont replace Perl 5. Lets show them its safe to come back and that there are LOTS of goodies to find when they do!

That said I want to take it even a little further. Perl has been proud of its backward compatibility, and well it should be! Code from many years ago runs well on very recent Perls. But what can be a blessing can also be a curse.

Perl 5 still supports many conventions and oddities from Perl 4 and before. There are some silly things like ' as a package separator, which most people neither use nor trip over. Then again there are some real things, like indirect objects notation, default script/handle encoding, new keywords.

Olof Stockhaus made a compelling case that Perl 5 might want to use the jump to Perl 7 to make some breaking changes. For example, among other things he suggests waiting for the p5mop project. However, I think I agree with Damien Krotkine in saying that that sounds like a great target for Perl 8 which could come soon afterwards. There are many things we wish for a future Perl, but until they are ready, there still could be some changes. I think Perl 7 could be a great oportunity to do minor, but important, breaking changes. Then let Perl 8 come in later with a MOP or a proper language spec. Either way, without a major version increase its hard to make breaking changes. Without a major version to increase, we have been stuck.

Now, some have suggested that we should drop the 5 from the version number and add it to the name. Therefore we would have Perl5 version 20.0 and Perl6 version 0.1. The problem is that this still doesn’t make the case that Perl6 won’t supplant Perl5. Those people like to make the case that Java dropped the 1 so that Java 1.6 became Java 6. However this comes back to my first point. They realized that 1.5 and 1.6 (or whatever was the actual change) was going to have major changes and should be a major release. Why they didn’t simply call it Java 2 I’m not sure. What I do know is that they weren’t trying to convince people to use Java1 version 6 when something called Java2 existed.

To comment on using the year as the version number I think there are two problems with that. First, it doesn’t really signify that Perl6 is not the successor. Secondly, what it does seem to signify is that the Perl community is going to abandon the stability that is one of its strongest suits. Yes I’m advocating some breaking changes above, but rather small ones, things that will help more than they hurt to fix. I don’t think we want to imply that there will be breaking changes once per year, we can still use feature to scope changes to Perl 7 through sub-major version releases. That said, breaking changes once a decade is probably ok.

Perhaps this is the root problem. The semantics of versioning are an important tool to inform the outside world what the state of Perl is!

Anyway here’s what I propose. Say we do implement the above, so now we have two languages Perl5 and Perl6. However, since Perl5 version 20 is mostly compatible with Perl version 5 perhaps it could be allowed to keep the name. Then it seems odd that Perl should jump versions from 5 to 20, so why not just move to the next available number 7. So now we have Perl 7 successor to Perl 5 and Perl6 a new language.

This does not signify the death of Perl 6 but the life of Perl. It just so happened that Perl 6 became its own language and started its own branch in the Perl family tree. Now that we know that the success of Perl 5 and Perl 6 no longer depend on each other, why should their version numbers?

After the break, my initial list of realistic goals for Perl 7

Edit: MojoCMS has been renamed to Galileo and released to CPAN. Enjoy!

Over the holiday break, I decided to have a little fun learning some things about the web. I usually get my Perl fix through science, but several upcoming projects might have some web involvement; so I thought I should brush up. The following are some reflections on that experience.

The task I set myself was to make a micro CMS (it is currently named MojoCMS, but I’m not sure I like that), leaving most of the heavy lifting to freely available Javascript libraries. I didn’t think I would be especially good at writing the actual interface, but rather the routing and functionality would be my task. In a strange way, the result was a kind of nostalgic Perl experience; Perl was the glue in my project again, not the main/only language involved.

I used several great libraries, jQuery of course, jQuery-UI for a small part, HumaneJS for notifications (works great for websocket responses!) and PageDown for a real-time markdown renderer. FYI, PageDown is the editor from the StackOverflow team. These projects make life much easier, I can’t imagine writing that kind of Javascript by hand!

I must say, Javascript still eludes me. I can parrot it, but I’m sure I’m not doing it correctly. I think the problem lies with its dependence on the HTML/browser that is running it; the odd way that the language doesn’t have a use command, and that “page”-globals can be used, still feels odd. I can definitely see the need for jQuery, but that adds even further cognitive dissonance. Anyway, I think most of this is my shortcoming, not its.

HTML5/CSS3 on the other hand is brilliant. Its easy to make the markup do what you mean without too many machinations. Of course I pull in some libraries for that too, namely Bootstrap.

Back to the Perl of it though, I must say I have high marks for Mojolicious, for many reasons, but the highest are for Websockets! Now I know Mojolicious didn’t invent them, but it makes them easy. Using Websockets I was able to make the “save page” and “update main nav” windows save without reloading. That was rather cute and feels modern.

The biggest point I want to make (long ramblings aside) is my most recent addition: DBIx::Class. I’m a scientist, not a database admin. I have setup some PHP CMSes and have used mysql just enough to get them started; terrified the entire time. So much so that I started my CMS project with the idea of using DBM::Deep for as long as possible. Soon enough though, I was nesting hash-keys three deep and wishing I had objects; if I hadn’t needed persistence I would have reached for Moose long before.

I investigated KiokuDB, and while I had some hope for it, I think I would need someone sitting in on the setup process with me. Then I remembered, throughout YAPC::NA along with the list of my favorite Modern Perl modules, everyone else adds DBIx::Class. Ok they can’t all be wrong. They weren’t. Sure the syntax is a little different from Moose, but its not that hard. The payoff for me started even before running the site, in the deploy command. With a simple script I can create all the necessary tables and inject sample content without ever needing to write SQL! After this the ORM quickly and easily replaced the DBM::Deep vestiges throughout the code, and just like that I have readable, OO structures for users, pages and the menu configuration.

Anyway, if anyone wants to play with MojoCMS (or suggest a better name!) feel free. It is still very rough but I may try to see it forward a little further. Passwords are stored in the clear for now, so be careful! But this is my next task. After that and some other work on users (like being able add them through the website!) the thing might even be able to host a small site. Not bad for a one-week side project!

Now that the official YAPC::NA 2012 talks are making their way online I wanted to post links to all of the relevant material for each talk.

Thanks again to all of you who attended my talks and asked great questions. To those of you watching for the first time, please ask me questions, I will be happy to help (if I can) or discuss my methods/paradigms/simulations.

Baby XS to get you started

A primer for writing XS for people who know Perl and a least a little C.

• GitHub repo (inc PDF of slides)
• YouTube
• SlideShare

Modeling Physical Systems Using Modern Object-Oriented Perl

In which I show off my object-oriented paradigms for simulating physical systems; forces and physical objects are just objects, interactions are mediated by coderefs. I also give a simple example and show off some of my real-world simulations.

• GitHub repo (inc PDF of slides)
• YouTube
• SlideShare
• GitHub repo for my real-world simulation

Further, the source of these talks serve as a decent example of using LaTeX/Beamer for making presentations. So check that out too. :-)

Good morning Chicago. I’m back from a wonderful trip to YAPC::NA 2012 and while its nice to be home, I’m really sad that the conference is over. It was my first YAPC and I’m sure that it will not be my last.

Before I get to my reflections, I want to say that the job that JT Smith(blog post picked because I like it), the MadMongers and the many volunteers and UW-Madison folks who made it all happen. I was so well organized and run, I’m astonished.

Moving on.

I started out a little starstruck. I haven’t been a programmer for that long (my first Perl scripts are dated 2009). I haven’t been around to watch the community develop, so I learned the names of major Perl contributors/community members from the CPAN modules etc. In my mind they sprung from the earth as fully-formed Olympian gods of programmers. It took until about lunchtime before it struck me that they all were really down to earth people and I started talking to as many people as I could. Pretty soon I was having lunch with Andy Lester, Sinan Unur, and Gabor Szabo among other perhaps less well-known, but no less interesting Perl users. I love that this conference gave me these opportunities! SawyerX is totally awesome! MST is hilarious (though I didn’t talk to him in person) and Larry even stopped by our impromptu “Perl Scientists” gathering the first night. I’m sure I’m missing some names that I could drop, but I think you get the point: the big names were there and were as great of people as they are coders/contributors/teachers/etc.

I learned some great things too. I saw Devel::NYTProf in action for the first time and I think I might see if it could speed up some of my more, lets say, “interesting” scientific scripts. I am going to add Data::Printer “awareness” to some of my classes, and I’m going use the concept of a specially-named-semi-private-method-for-awareness the next time I think about the interoperability of different modules (if that doesn’t make sense, look into the _data_printer filter option). I think I have a better handle on when to use Moose Roles. I’m excited about new Perl features (__SUB__++).

I even have a new project; a tryruby-like web-based “Try PDL”, which might-or-might-not use vti’s showmetheshell and might-or-might-not be hosted on Stackato (and even if it isn’t I learned about LXC for sandboxing). Would more people use PDL/Perl for science if they could play with it before installing (the much improved, but still rather tricky to install) PDL module itself? I also need to make more websites for my modules (as mentioned in the last lightning talk session).

Finally, I gave two talks myself and was so excited about the attendance and reactions. The first was about my reseach, and more specifically the methodology that I use to make modeling easier both for the module-author (in this case: me) and the script-author/user (me and others). The room (235, the best room IMO) was full, probably about 30 or so people and they seemed very interested and engaged. Reini Urban even paid me a great complement in front of the whole room (!!!). The full model is available here.

Then I (immediately, though in another room) gave a how-to-get-started talk on XS. I’m glad I didn’t have more time to worry, because the room was bigger and full too, I don’t want to guess how many attendees there were, but it was a lot. I really hope people learned and will be able to use this knowledge as well. The feedback I got was very positive.

So thanks to all of you who attended my talks! You really made my conference even better! The links above contain the PDFs of my slides. If anyone has any questions or would like to hear more about my research, please contact me. (As an added bonus, the LaTeX-Beamer source used to make them is also there! The PGF/TikZ/Beamer modules have started a similar renaissance in the LaTeX world like Moose etc have started in Perl FYI.)

I guess this has become a really long post, but I had such a great time. I can’t wait to relive it, and get even more knowledge (!), once the the recorded talks are posted.

Cheers!

Part of the reason I love Perl is that it has full closures. It makes it really easy for my scientist brain to think of code references as equations. This is why I try to make my scientific software think this way too; PerlGSL is built on this concept. Its especially fun when this allows me to nest functionality to get even more complex behavior.

In this following example I find the Gaussian width needed so that the integral over it on a certain range has a particular value. Yes this is an easier example, but if the function f were more complex, this code would be no worse.

Hopefully this shows some of that power, and some of the reason I am working on PerlGSL.

I will be going into more detail on similar concepts at one of my talks at YAPC::NA later this month. See some of you there?

I have some news coming soon about Alien::Base but to avoid burnout, I’ve spent some time in the last few days playing with some things that are new to me. I enjoy doing this any time I’ve spent too much time on one project.

While I have spent some time using Mojolicious it has always been to hack together a quick UI for some code, rather than pulling out Tk. I never have really taken the time for pretty-fication, nor for any kind of interface logic.

I have a friend who thinks highly of my programming abilities and has recommended me to another of his friends to put together a website for a startup company. While I could put together a Joomla or Drupal site in no time, I thought I would investigate a Perl solution. I know that there are a few Perl CMSes out there (WebGUI 8 sounds interesting), but I wondered if I could hack something together myself.

Here’s the problem, I know Perl, thats about it. I don’t know much JavaScript, CSS or even HTML for that matter. Forget about DOM, and I’m no graphic designer.

However, surprisingly, Mojolicious, along with Twitter’s Bootstrap for building page elements, has made this really easy. I don’t know if I can sell it to a client yet. But maybe by the next client, or the one after that, I can offer a website as a Perl/Mojolicious/Bootstrap/PSGI app!

I love open source!