I am planning to write a small guide/comparison of the different ways to start a process in Perl and while thinking about all the ways i do know, became certain that:

1. my knowledge about the ways i know is not complete
2. there will be ways i do not even know about

Thus i am turning to the Perl community for help. Please tell me what ways you know to start a process in Perl and all of the permutations it can go through. :)

In my previous post oalders (of Metacpan) asked me about how working on Windows influenced the process and in answering him i got a bit more wordy, so here's another post:

Video is a bit easier on Windows because, even if many of those are very amateur, it has a wider spread of tools available. However, that said, i still needed a bunch of linux tools (that i could luckily run on windows).

First issue was to free the stream from the Silverlight player. As mentioned in the previous blog post, lots of people helped with that.

Next was getting it dumped to the hard disk into a file i could actually play back afterwards. Windows Media Player couldn't, Mplayer crashed and only VLC finally worked. The command lines i saw didn't help a lot, but it led the way because it did produce some kind of file. By fiddling with the GUI i could get it to dump a usable file. And after converting the gui settings back into command line instructions (after cursing VLC's terrible documentation) i was able to perform the dumping with scripts. The command line instructions ended up being:

qw[ --sout-keep --sout "#transcode{vcodec=div3,acodec=mp3,vb=6000,ab=320,deinterlace}:duplicate{dst=display,dst=std{mux=asf,access=file,dst=$dump_file.asf}" ]

The "keep" option is important. When the stream was cut off at the venue, i.e. by shutting off a camera, VLC would try to reconnect to the stream and in the process overwrite the data it had dumped with an empty ASF header file. The keep "keep" option prevented that from happening (but not before a lot of data was lost). Another issue here was: While streaming and dumping the volume controls of VLC did literally nothing. Windows 7 helped here by providing me with volume sliders per process.

Next was converting the raw ASF stream into something that could be distributed. No tool i had could actually read it. At first i tried ASFTools, but that only resulted in a very desynched stream. This is caused by the fact that ASF has non-constant FPS. This doesn't mean it changes sometimes, but literally all the time. Hobbs helped me later with the conversion and said the file was apparently at 1000 FPS. So ASFTools was useless for the conversion, but it helped me with one thing: The asf files were unseekable, so short of watching the entire thing i could not tell what was actually dumped. ASFTools was able to fix that without recompression.

Hobbs contribution here was also immeasurable. I was entirely stumped with the conversion. None of my tools could make sense of the ASF files, even though they could play them. He worked out and provided me with parameters for mencoder to reencode them into AVI format, which ended up, after some fine-tuning, to be this:

mencoder -quiet -oac mp3lame -lameopts q=1 -ovc lavc -lavcopts vcodec=mpeg4:vqscale=3 -ffourcc DIVX -ofps 30 -o %1.avi %1

After that the lion's share of mental work was done and what remained was mostly menial work: Cutting the files into pieces and uploading them. Here's another part where Windows was really useful. VirtualDub is a free, open-source and lightweight tool for video processing. It also makes it extremely easy to cut videos without reencoding them by making it easy to navigate between keyframes, select regions and dump them straight to fily with Direct Stream mode. This took considerable time because i needed to go through all the dumps, find appropiate start and end points for the talks and figure out exactly what talk it was and what its title should be.

Last came the uploading to Youtube. I had luckily recently upgrade my internet connection, so i could upload at ~300kbyte/sec. However this also meant that i had uploaded four 50-minute videos before Youtube told me that it rejected the files for being too long. However, this restriction only required a validation of my cell phone number, which only took a minute all in all. I did need to reupload those videos again though. After that i dumped them all into a playlist (which i had to nuke at some point, because Youtube will happily add videos to playlists multiple times) and sorted them in some kind of chronological order.

And now you can watch them. :)

One last note: After i had uploaded many of the videos to youtube already, JT sent me an email to ask me to discuss a small matter relating to the videos and also mentioned that while i had not obtained permission, he would allow me to leave them up. I was a bit unhappy about it, but in retrospect he was correct. I did not think of it at the time because i was too busy figuring out the technical details, but i should have asked. However, please don't take this as discouragement in case you ever think of doing this for another con. Con organizers will love you to death if you offer to do this for them. Just make sure you let them know in advance and ask permission, because sometimes they have to deal with legal issues involving third parties.

I wasn't able to attend YAPC::NA 2012.

Yet i was able to attent YAPC::NA 2012.

"Huh?" you say.

Simple, i was what is called a Remote Attendee. With #yapc on irc.perl.org, the con had an excellent and well-frequented IRC channel where both people that stayed at home, as well as physical con attendants mingled and talked about god, the world, con organization, the talks being given and software in general. Everyone was able to chat about the talks because the con orga had managed to make live streams available for four of the five con rooms and that in excellent quality.

So thanks a lot for JT and the rest of the YAPC::NA staff for a job extremely well done!

The only dark-grey mark: The venue insisted on using their own system, which streamed ASF via a Silverlight player. People could use Moonlight, but a good number of people refused that on principle. Enterprising perl mongers quickly figured out how to play the streams with VLC and also how to dump them to disk. On Friday, the third day, i started using these to capture on my windows machine and despite some setbacks via VLC, i was able to grab a bulk of the talks on that day.

All of these videos are now up in this Youtube Playlist. So if you didn't get to see any of the talks, or missed one you wanted to see, have a look, maybe it's in there. And if you haven't seen Perl and CPAN or the Surprise Guest Speaker talk you should go see them now.

As well, you might want to check back later, as i'm still working grabbing and converting more backlog. It won't be complete coverage of the con, which will be done later by Presenting Perl, but it should tide you over until then.

On request i can also make the raw video files available that i got. You can grab me via my CPAN email or as Mithaldu on irc.perl.org.

Recently an article was published that claimed Perl usage in websites had dropped below 1%. For a Perl developer this seems ridiculous on the face of it. As it turned out, it was ridiculous, since the error margin was 17.6%. Meaning that for 17.6% of the sites they surveyed they could not even detect that they were using Perl. And they only survey the top 1 million, out of the 175 million active sites Netcraft reported. So the validity of these numbers is highly suspect.

So, all is fine? Catastrophe averted? Heh, not really.

Here's the issue: To a Perl developer this might seem ridiculous. Most of us are aware of how many websites use it under the hood. But to the average person it seems perfectly reasonable. The suffixes ".php" and ".asp" are ubiquitous and lets everyone know what the website is running on. Ruby and Python operations are proud of using their languages and flaunt it. To an average person it might even seem like there are more Ruby and Python websites.

Yet Perl is practically invisible. It drives a lot of sites, businesses and livelihoods worldwide, neither of which make any indication of the tools they're using. Let's take a look at a random sampling:

These are sites that run on Perl, but to look at them you'd never even know it. Perl has advanced route dispatching mechanisms and its great modularity makes it possible to write an entire website as a single application. In fact, for most modern Perl web frameworks it has become customary to run web applications as self-hosting services to which a web server connects via a network port. Perl has made it very easy to design one's url structure exclusively around the data one serves, which has made it entirely impossible to tell for most Perl-driven sites that they actually are Perl.

Now, all of these sites have reasons to be proud of the tools they use. Some of them indeed are very proud, booking.com for example has donated 150.000 USD last year to Perl development and others on the list have donated smaller amounts as well or are supporting Perl conferences and spend a lot of money to have a presence on Perl conferences. Those on the list who are hiring, are also hiring Perl developers, some of them quite prominently. Even others on the list are websites created for the express purpose of serving the Perl community.

Yet none of them even mention Perl on their frontpage or even link to perl.org. I know Perl doesn't have nice and easy promotion buttons like Python or PHP do. But a simple text link does the job just as well. Just a simple "Built with Perl" in the footer or a sidebar would show that you care about Perl, would make it easier for job seekers to notice that you're looking for Perl devs and would raise the visibility of Perl and make it a more appealing choice for students looking to learn a dynamic language.

I do not believe this is done out of any maliciousness and that it's rather just a simple oversight, something that was never considered. As such i've sent a small email to each of those sites, asking them to consider this issue and to please add a link to perl.org or to weigh in on why they would not do it. Maybe they are issues that the Perl community can fix.

Similarly, if you own a site yourself that runs on Perl and does not link to perl.org:

Background

I like the Perl debugger a lot. I use it daily, since understanding code by seeing it execute is much better than guessing at what it does based on its API. I do however seldomly use the actual command line debugger module, but instead use the Perl debugger built into Komodo IDE because it streamlines a lot of the busywork that is necessitated by having a debugger bound to a terminal:

• It automatically shows the variables in the surrounding lexical scope instead of making me type x $var<ret> every time i want to see my variables.
• It allows me to step in, over or out of statements/subs by clicking, which is a lot faster than typing s<ret>, n<ret> or r<ret> a lot.
• It allows me to have the debugger run to a certain line in the current or also other files by just setting the cursor on the line and hitting the shortcut for "Run to cursor".
• It allows me to define and store breakpoints based on the results of perl expressions without actually altering my source code.
• And more.

Problem

All is not perfect however. Both Komodo IDE and perl5db.pl lack nice interfaces for doing the opposite of a breakpoint, which is ignoring statements. Komodo IDE has flat-out no interface for that. The core perl5db.pl allows the user to define a sub DB::watchfunction() which is executed in the debugger's scope and can skip out, but that's a lot of busywork and means I'd have to specifically write the sub for all of my code.

Why is this important? Consider Web::Simple's role Web::Dispatch::ToApp. This is executed a lot during the dispatch process, which is also where i do a lot of my debugging to make sure my routes are solid. That means whenever a piece of code goes $obj->to_app->(), i have to step into to_app in order to be able to step into the actual call sub of $obj. Another example would be Moo's Sub::Defer::defer_sub::ANON, which is invoked on almost all Moo attribute accessors. That kind of thing is annoying and a waste of time.

So, what i needed was a module that can be loaded in my code, will be given some import parameters and will cause both the Komodo IDE debugger as well as perl5db.pl to skip over statements based on package or fully qualified sub-routine name.

If you don't care about the implementation, you can skip to the end of the article now, where i show the solution.

Implementation

Thanks to an article by brian d foy (hope i got his name right) I found that the Perl debugger modules are actually fairly simple: "Under the -d switch, perl calls &DB::DB before each statement" This is also documented in more detail in perldebguts.

Armed with that knowledge it's simple to write a function that augments the current DB::DB() with a bit of code that interrogates the caller before heading into the debugger proper. We can't just directly call the old DB::DB() function, since it relies on the stack level and makes heavy use of caller(). So, a simple replacement that stores the old function and uses goto &{$ref} should do the trick:

    my $old_db = \&DB::DB;
    *DB::DB = sub {
        goto &{ $old_db };
    };

Sadly it turns out that the goto makes the whole thing go out of the protected debugger scope in which DB::DB is not executed for every statement; resulting in something akin to a fork bomb. So before we can actually do interesting stuff, we have to add a bit of code to skip out when we're being called from inside DB or any other DB::* module:

    my $old_db = \&DB::DB;
    *DB::DB = sub {
        my $lvl = 0;
        while ( my ( $pkg ) = caller( $lvl++ ) ) {
            return if $pkg eq "DB" or $pkg =~ /^DB::/;
        }

        goto &{ $old_db };
    };

I think it should be possible to munge with the stack directly as well, so we can just use return $old_db->( @_ ), but my Perl-fu is not that good and Sub::UpLevel looks fairly daunting. (Suggestions welcome though.) One would think that doing it this way would make the debugger a lot slower, but honestly, i've been using it for a while now and couldn't notice a difference.

So, now we have a piece of code that acts as a pass-through to the debugger and doesn't blow everything up. That means we can add code to actually do things, but this turns out to be fairly trivial.

    my %pkg_skip = (...);
    my %sub_skip = (...);

    my $old_db = \&DB::DB;
    *DB::DB = sub {
        my $lvl = 0;
        while ( my ( $pkg ) = caller( $lvl++ ) ) {
            return if $pkg eq "DB" or $pkg =~ /^DB::/;
        }

        return if $pkg_skip{ (caller)[0] };

        my $sub = ( caller(1) )[3];
        return if $sub and $sub_skip{$sub};

        goto &{ $old_db };
    };

There we go, a working piece of code that makes the debugger skip over statements in certain scopes or subs.

Solution

With a bit of parametrizing and cleaning up of this, the result is DB::Skip, a module that you can load in your code when you want to augment the debugger to skip statements. The use is extremely simple and it's made debugging a lot more pleasant for me already:

    use DB::Skip (
        pkgs => [
            qr/^Method::Generate::/,
            qw(  Sub::Defer  Sub::Quote  Web::Dispatch::ToApp  )
        ],
        subs => [ 'Web::Dispatch::MAGIC_MIDDLEWARE_KEY' ]
    );

Perl Tutorial Hub

And now back to your scheduled rant:

• PREFACE
• THE PROBLEM
• Solutions
• Content Creation
• Outreach
• SEO
• QUESTIONS

PREFACE

On monday a blog post made the rounds that claimed "Perl users were unable to write programs more accurately than those using a language designed by chance." Naturally that is not the entire truth. As it turns out they "trained" their test subjects using examples of code that they considered representative of what a user would find when trying to learn Perl from the internet. That code looked like this:

$x = &z(1, 100, 3);

sub z{

    $a = $ [0];

    $b = $ [1];

    ...

    for ($i = $a; $i <= $b; $i++){

    ...

    if ($d > $e) {

        $d;

    }

    $e;

}

Naturally the outcry (including from me) went: "But that is Perl 4 code, of course the test subjects had a friggin hard time with it!" But were they wrong in choosing that example to teach with? Was that representative of what a complete perl beginner would find online?

THE PROBLEM

Turns out, that yes, it was representative. The code above represents exactly the very first thing found when looking for Perl learning material.

How so? In a post yesterday i asked for help in finding out what a perl newbie would try first when wanting to learn Perl. Google Trends is a great tool, as it shows comparatively how often certain search terms are used. And it turns out that among the highest performing search terms there is one that is at least twice as often used as any other search term:

And quite frankly, the results for that search suck. Incredibly so.

The first result, a site by the University of Leeds, in the UK, carries this warning at the top: "This tutorial was written in the early 1990's for version 4 of Perl. Although it is now significantly out of date, it was a popular source of information for many people over many years. It has therefore been left on-line as part of the historical archive of the Internet." 21 years later, in 2011, it is still the very first thing Google brings up when looking for perl tutorials and that is exactly what newbies world-wide do first.

Make no mistake though, it is not just the first result that is terrible. Result two ( perltutorials.org ), features Perl 4 code as well and result three ( tizag.com ) attempts to teach "PERL", also using Perl 4 reminiscent code. Search result four is a highlight, in that it is Gabor Szabo's excellent fledgling Perl tutorial series with videos. But the very next result goes back to 2000, with an article on perl.com, which already features 5.10 and nicer forms of `for`, but still eschews talking about strict, warnings, or even using `my`. After that comes Learning Perl, which is a bit barebones, but features a nice modern Hello World. Yet right next up is a link to a free online copy of the book Beginning Perl, which might sound nice, but is anything but, thanks to being the edition from 2000, which teaches Perl 5.6.

I could go on, but it doesn't get any better. Only worse.

If you've wondered why the public perception of Perl is that of an old language, this is the reason. Our public face to anyone new to Perl and wanting to learn is dominated by material from the turn of the century, or earlier.

SOLUTIONS

Now, i honestly do not know how to best solve this, but i can bring up some ideas.

Content Creation

Now, as far as i am aware there is no current, definitive and in-depth Perl tutorial anywhere. Please correct me if there is one.

Gabor Szabo has made a good start, but it's only a start. Right now his series is tiny and he is only one person. Another good attempt it learn.perl.org, bug again, too small in scope. It provides a few bites, a few examples, basically just a taste of Perl and then tells the peruser to look at other sites. These approaches suffer from a few flaws in that they fail to properly capture and provide for the newbie.

So, we need a good modern Perl tutorial. Not just books. We need something in the style of the Leeds University document, but with all the anachronisms filtered out and current knowledge inserted. Something like the Python Tutorial. And most importantly, it needs to be a community effort. It cannot hinge on just one person.

Possible options i see are to either create a wiki on a domain like tutorial.perl.org/com and take pains to ensure that editing and updating it is easy and effortless. Alternatively it could be done by ways of a github repo, possibly even within the perl core repo, which is then exported to HTML regularly.

Using a structure like this, the community at large could concentrate on writing the textual content, while people like Gabor can still contribute videos. It could be a very nice mix of the python tutorial and RailsCasts.

Make it good enough, publicize it and Google search results would right themselves on their own.

Outreach

Another option would be to reach out to the maintainers of those sites and convince them to take down their sites or at the very least add links that prominently point to more modern content. Alternatively they could be asked to cooperate with the Perl community in making it possible to change and update the contents of those sites.

Then there's also the matter of the people linking to those sites, outreach could also be done to them by ways of finding out with www.backlinkwatch.com who links to them, and then convince those sites to remove/change their links to newer materials.

SEO

Failing the above two, there are still measures that could be taken to aggressively alter Google's view on what's relevant to perl tutorials. This could be in the form of burying results we do not care about, or promoting results we do care about. It is in my eyes a last resort and might involve money expenditure, but given the circumstances i do not see it as entirely unwarranted.

Edit: A suggestion from Reddit: "the new +1 feature for search results may come in handy. Get the entire community to +1 the same link."

QUESTIONS

Some questions still going through my head:

• Are there perl tutorial sites i do not know about that have good and copious content?
• If so, how could we effectively advertise them, so Google picks up on them?
• Are there other options i did not think about?

And with that, i close this entry and leave it to you to ponder and provide feedback and most importantly: Help repair the image of Perl.

Right now i'm concentrating on what kind of search term a newbie would enter in Google. Luckily Google trends help a bit there, allowing me to compare between various search terms.

These are the search terms i came up with or was suggested:

perl tutorial 1.00    perl cgi 0.79    programming perl 0.45    perl examples 0.37    perl learn 0.10

Please help me and see if you can think of any other search terms that would be more often used than "perl tutorial". Thanks in advance.

ActivePerl's Jan Dubois has kindly applied a patch to DBD-mysql 4.020 on their servers, so you can just do this:

Original post:

This a warning to Win32 perl devs using DBD-mysql.

The versions 4.019 and 4.020 are broken in a rather subtle way: All SQL errors are silently ignored and in fact not even triggered, no matter what RaiseError is set to or what kind of error happens. As an example, this code will run without a complaint:

Various messages have been sent to the maintainers of DBD-mysql, including a proposal for a fix. However, implementation of a fix and release of a new version will likely take some time.

In the meantime downgrading is the only viable option, but sadly the ppm client does not make that kind of thing very easy. As such, this is the command you will need to execute to downgrade your DBD-mysql:

Note that is compiled for Perl 5.12. By changing the Perl version part of the url you can also get it for 5.8 and 5.10. For 5.14 there is no ppm available though, so if you're using that you will have to downgrade your entire Perl or try your luck with the 5.12 one.

So they stated dutifully adding `| html` all over the place. But that's stupid. It's playing whackamole because this sort of issue will keep cropping up again and again, often only to be noticed after it's abused.

However now it's really easy to fix this by having the html filter applied to all tokens in a template:

To break this down:

This is your typical TT view for your Catalyst app. It uses on Catalyst::View::TT as base class, but defines v0.37 as a minimum. This version (released today) supports the CLASS parameter, which allows you to set the class to be used to create the TT object.

That makes it possible to use Template::AutoFilter instead of pure Template. This is great because T::AF automatically applies a filter (html by default) to any unfiltered template tokens. (Excluding a bunch of control tokens that either pull in other templates or have no output.)

What this means is that with this little code snippet above you have a view that automatically converts any user input so all html elements are rendered as entities and displayed plainly to the user instead of being interpreted by the browser. Of course, you'll want some parts to not be filtered, and that's easy to do, since you can either apply some other filter, or just add `| none` to skip the auto-filtering.

This means there's no way for an attacker to inject html or javascript into the template-driven parts of your website unless you explicitly enable it.

However, one thing stands out here: It does not seem to be clear to either party exactly why the party is saying what it is saying. And when the realization arrives, some realize that it should be fairly obvious and stop arguing, but do not make their realization public, leaving others to make the same mistake.

As such, i'll try to put it into the most simple words i can here:

The existence of Perl 6 means that for the decision makers in IT, the managers and CEOs who are not even programmers, Perl 5 appears to be obsolete.

This is not a language issue, this is not a feature issue, this is not a community issue. This is entirely an issue of marketing Perl 5 to the people who control the money, because they decide which developers to hire and what languages to train their developers in and which language to use in their next project. Many of them will not make the decision based on how shiny Moose, Dancer or Mojolicious are, or how many modules there are on CPAN. They will make the decision based on gut feeling and on how well they can sell it to shareholders.

The name Perl 6 implies to non-technical people that it is newer and better than Perl 5 and that Perl 5 is abandoned and a waste of time and money to invest in. There is a lot of vibrancy and activity in the Perl 5 community, with new tools and toys springing up weekly and improvements being made to the core regularly, but this is only visible to programmers, people who are on the ground and working on it.

People who ask for name changes do not do it so they can effect great upsets in the Perl development. Perl 5 has had a great many upsets since it was first called Perl 5.

People ask for name changes so they can show the rest of the world just how alive Perl 5 really is.

Perl debugger is awesome. Komodo IDE makes it even more awesome and can play with the guts of a Plack web-app while the web-app is running on a web server. And now all of this now works on Windows too, by updating Plack and applying a tiny patch to Komodo.

Recently I wanted to do some Dancer development on Windows. For me web development means making use of some fairly powerful tools in order to remove delays and increase the turnaround speed. Sadly the state of the stack under Dancer, as well as one of these tools hampered my efforts, leading me to fix them. In following I will lay out the tools, the needed fixes and my web development workflow.

My favourite tool in the vast toolbox Perl provides is the debugger. Simply adding '-d' to the command line permits stepping through a program statement by statement, halt at any line, halt only when certain conditions are fulfilled and most importantly: It permits examination of the content of any variable at runtime by simply issueing 'x %var' and even to execute arbitrary code at runtime to change variable contents, thus allowing more complex analysis on variable states or change on the program execution flow.

In short: It allows full insight and control over any Perl program.

However, as it is merely a command-line tool, all these powers are not as speedily accessible as they could be. To find out when a certain variable is changed in a program it is necessary to reissue the x-command after every statement. In order to run a program under certain conditions it is necessary to rebuild them on the command line. This can be fairly time-consuming when trying to reproduce the circumstances of a web-application, as %ENV needs to be adapted, as well as cookies provided.

Both of these problems are solved by Komodo IDE.

It is able to start Perl programs in debug mode and wraps UI elements around the debugger controls that automate certain things. It recognizes lexically close variables, provides them in a list and automatically issues the x-command on them at each step, making it possible to literally watch the program state change while stepping through it. It also provides alternate views for global variables, special variables like the contents of @_ and most importantly watches on arbitrary Perl expressions. This makes debugging a process much faster than working with the debugger manually.

Secondly it provides a network interface that programs running in debug mode can connect to, permitting remote debugging. Adding a '-d' to the hashbang line at the start of a Perl program and having the %ENV configured to load the Komodo-provided debugging library (simply a module in the vein of Devel::Cover, NYTProf, etc.) makes it possible to have a program running on a web server in London connect to Komodo IDE on a workstation in Berlin, enabling all the aforementioned inspection and control.

For me this means that my development environment consists of any web server with a few configuration settings to tweak %ENV so the Komodo debug library is loaded. Perl programs are run in CGI mode through Plack, so I can toggle them into debug mode whenever I wish without needing to restart the server. Finally, Komodo IDE on my development platform to accept any debug requests sent by the Perl programs started by the web server. In production things are run with the FastCGI handler, but even there i can switch to the CGI handler at any time if i need to look into the guts of my application.

This is where the first problem occurred: Trying to run Dancer through Plack on Windows resulted in spurious server errors and corrupted or truncated pages when using the CGI handler. This was caused by Perl's (unneeded and pointless) newline mangling on Win32, since it meant the HTTP headers sent by Plack were wrong and binary data was corrupted. After figuring this out, a patch issued to miyagawa was integrated in less than two day and the issue fixed in Plack 0.9977.

Then the second problem became obvious: Fixing the newline mangling necessitated a change that Komodo's debugger library didn't know how to deal with, resulting in the mangling staying activated. Luckily it was easy to fix and a patch sent to Activestate, which was accepted for Komodo IDE 7. If you do not use the nightlies or alphas, you can also apply the patch yourself: https://gist.github.com/960360

Simply by making sure that you have the latest Plack and the above patch applied and you can do ridiculously fast web development with Komodo IDE, using powerful and convient debugging features. (And all of the above really applies for any framework using Plack.)

Thanks to p5p, Miyagawa and Activestate for making such tools available and even more so to the latter two for acting fast on patches.