Joyent

Six months ago I went to create my first instance and was disappointed. The selection of Linux distributions was just Debian and CentOS, both of which have old versions of Perl. Now the Perl programs I plan to use and the future ones I write are for new versions of Perl. I didn't even think about this until I had a running Debian instance and had a program error out on me. I had used the s///r option that was added in Perl 5.14 and Debian has 5.10, what a bother. I install perlbrew and Perl 5.16.3, then all the cpan modules I need. By the time this is all over I have a newer Perl but dependency problems from Debian's packages being older reared its head again.

Now the Debian users will tell me to switch to testing or unstable or add these other reposities. That is all fine and dandy but when the dependency is gtk3 and installing that breaks existing applications because gtk2 and gtk3 are not designed to be installed side by side, this poses a whole new kind of problem. I wanted to test some scripts I wrote using Gtk3::WebKit for headless automation. Setting that aside I spent a month trying out different applications and testing the network. There were no power outages, and no network issues at all.

Recently Joyent has added two versions of Ubuntu and two of Fedora. The newest of all the options available now is Fedora 18 and I recommend going with that. I used Fedora 17 (Beefy Miracle) extensively to test another cloud hosting provider and having Perl 5.14.4 as the system perl was nice. I did not have to install perlbrew and build another perl to get all the new things. Fedora 18 ships with 5.16.2.

The billing system is just missing all kinds of features. You cannot view your bill based on current usage, which is a feature they plan to add. You cannot load a prebuilt image onto Joyent because they have no import capabilities. The closet you can get is to build up an instance and then save it for later, for a continual usage fee. Some of the download links on their site point to documentation that does not exist.

Would I recommend it?

If you are planning on setting up a server and leaving it running for long periods of time Joyent is a good pick. When trying to spin up instances quickly and frequently the lack of import features for operating system images makes Joyent a bad fit for that use case, hence longer jobs. A longer job only has to be setup once and that time is a small fraction of the whole. As for the missing features I mentioned above they would be really nice to have but do not stop me from using Joyent effectively. Using the API with Fedora 18 would make spinning up instances faster and easier but the problem of out of date packages still persists. Being able to load a custom external image is the way around that problem.

Joyent use it, but understand it still needs some polish.

A little search-fu later and I am disappointed. I find broken links, closed source pay applications, (yeah like I am going to pay for a closed source application to access a public API) and a stack of how to blog posts consisting of nothing more than 'Download this program for Windows and click here, here, and here.' I hit up a few friends on irc and they all strongly recommended youtube-dl, a Python script that has command line options for getting playlists. Let me be clear when I say strongly recommended, I was told about how they had used it for years with no problems, how it was the go to script for downloading different items from youtube, etc... My expectations are now set pretty high considering the high praises my friends put forth meaning, I expect it to work for the problem I have.

youtube-dl the disappointment

I start by going to github.com/rg3/youtube-dl to pull down the latest version and I notice there are 193 Issues and 8 Pull Requests currently open. I clone the repository, set it up and play with it. I build up the following command to fetch just the playlist from youtube.

./youtube-dl https://www.youtube.com/playlist?list=FLNj0f88cIMwioXOKJE89Lpw --simulate --get-url > music_vids.txt

Now youtube-dl defaults to trying to download the videos when I only want the urls, hence the simulate flag. The above combination did not work nor did the various combinations I tried out, including adding --ignore-errors. Instead I ran into the following problems:

• youtube-dl is slow. It took 2 minutes 17 seconds to get 36 video urls from a list of over a hundred before failing out. I expect this api call should only take a few seconds.
• youtube-dl fails when a video in the playlist has been taken down due to DMCA violations. The application just exits with the error message and does not process the rest of the list.
• youtube-dl fails when a video in the playlist has been deleted by the author. Fails and exit.
• youtube-dl fails when a video in the playlist has been blocked based on your geo-location. Fails and exits.
• The urls youtube-dl returns are for the CDN locations not the canonical urls. CDN urls are not guaranteed to be valid in the long term. At the very minimum this should be noted in the documentation.

I checked the 8 pull requests for fixes and nothing, same for the issue queue. By now I have spent about twenty minutes on research and trying things out. At this point I think it will probably be faster for me to write a program to get the playlists then to try and fix this application or keep looking for something else.

Build it yourself

The first step is to see what modules are on CPAN. Now I am going to go into a little more depth about how I research this to show some of the potential problems with our Perl ecosystem, the key one being signal to noise ratio. Finding quality in the mass of everything is not a problem unique to Perl but the Perl community has come up with some interesting solutions. I open up search.cpan.org in one tab and metacpan.org in another. I do this because the search capabilities of both are different, amongst other things. Metacpan's search is great if the keyword you are looking for happens to be part of the module name.

youtube-playlists more disappointment

On metacpan when you put "youtube" in the search field the first auto-complete is youtube-playlists a script that is part of the WWW::YouTube::Download module. It was last updated May 5, 2013, has zero bug reports, two five star reviews and zero failed test reports. The synopsis shows example usage that looks exactly like what I want. I install WWW::YouTube::Download with no problems and run 'youtube-playlists FLNj0f88cIMwioXOKJE89Lpw' and it returns 25 results only on a play list with more than one hundred entries. Looking at the POD there are no cli options to change the result count and looking at the code I see it is only pulling down the first page of results as xml with no mechanism for handling pagination. I consider hacking away at this program but there must be a youtube module that handles most of the work instead of just pulling in raw xml feeds and doing it all yourself. I am also surprised this program did not show up in the general search engines, only in the CPAN ones.

WWW::YouTube

WWW::YouTube has not been updated since July 28, 2008 and has one review which is one star. The positive points are there are zero bugs in RT and zero failed test reports. The single review raises multiple issues that I agree with based on reading the POD for the different modules WWW::YouTube provides. Therefore WWW::YouTube is not a good fit.

WebService::YouTube

Last updated Jan. 20, 2009, it has 4 open bug reports and zero failed test reports. Glancing at the POD I notice the following message: This module support only Legacy API, does not support YouTube Data API based on Google data protocol. Now to me in this instance Legacy reads as deprecated so why would I build something on that?

WebService::GData::YouTube

WebService::GData::YouTube was the next viable option last updated Nov 13, 2011, with 3 bugs, 1 five star review and six failed test reports and 515 passed reports. A quick search for 'playlist' in the POD reveals get_user_playlist_by_id() - Retrieve the videos in a playlist by passing the playlist id. Exactly what I am looking for and it installs with no errors. Here is my short program.

I call get_user_playlist_by_id() in my while loop so it keeps fetching blocks of 25 videos until the playlist is done. Videos that have been blocked or taken down can be skipped because their 'duration' field does not exist. The urls returned as part of $video have query parameters that are not needed, so I use URI::URL to pull out the host and path. The program outputs the playlist position number followed by the canonical YouTube url.

Other Modules

I also took a quick look at other modules to see what else there was. WWW::YouTube::Info and WWW::YouTube::Info::Simple do not have any playlist functionality.

The take away

I would like to think I selected the best module available but there could still be something better on CPAN or out in the wild. It is always a balance between time spent doing research and building it yourself if you can easily conceptualize the problem. I think far too often people get sucked into the research phase which can be quite fun. I use the bug count, last release date, pass/fail test reports, CPAN Rating, and the list of previous versions to determine if a module is mature enough for a deeper look to solve the problem I have.

There is no single search engine that encapsulates everything so just using one generalized option like Google is not enough. Searching CPAN directly is always worth while. Next stop would have been github, then bitbucket, and sourceforge.

Now the above is basically the first example in the POD and not all that interesting. The "fun" part was discovering that some of the processes were hanging. Digging in I find it is the other application I am calling that is hanging because the file it was trying to read is for some reason unavailable. This kind of failure is acceptable, just log what didn't work and move on. To clean up the stale processes I used Proc::ProcessTable to check for jobs that were running too long and kill them every number of processes configured for Parallel::ForkManager. The code looks like this now:

I tweak the $process_count based on number of CPU cores since these processes are CPU heavy but RAM lite. I had multiple versions of this application running on different server sizes for the last week to determine performance metrics for the future. I want to squeeze as much work out of a system as I can every second it runs. Yes I do love optimizing code and fast results but this is purely practical. Above I mentioned a list of 40,000 items. Since I started using this little application I wrote that source file is now a few hundred thousand items long since processing time is not terribly long.

This is how I feel now.

This is where I found out about Path::Iterator::Rule which was just released by David Golden. It works the same as Path::Class::Rule but returns strings instead of objects, which gives a massive performance boost. Path::Iterator::Rule is a drop in replacement for Path::Class::Rule so updating my programs required very minor changes.

With Path::Class::Rule my application took an average of 66 seconds per run. Now the Path::Iterator::Rule version only takes 5 seconds with the same input. A full minute saved on each run, it feels good.

I am reminded of a quote from Brad Frost's article Performance As Design in which he states "Good performance is good design" and while the article context is web development, I think it applies to any kind of application.

Initially I just had an array of patterns that I fed through qw/$_/ixms and it was slow. I had not considered using alternation because I thought it was going to be too slow. Perl 5.10 fixed this kind of problem but I was so used to how it performed before that I had not considered it since. With this new information in hand I created a new benchmark set to compare performance. Here are the old numbers:

./method_bench.pl
         Rate method4 method5 method3 method1 method2
method4 585/s      --     -0%    -35%    -40%    -40%
method5 586/s      0%      --    -35%    -40%    -40%
method3 898/s     53%     53%      --     -8%     -8%
method2 972/s     66%     66%      8%      0%      --
method1 972/s     66%     66%      8%      --     -0%

Method1 was copied into the new test as a baseline. I purposely added a solution that I know would be slower ‘regex_overhead’ just to see what would happen. ‘regex_assem’ show no performance difference if I use $regex3->re() or not.

Running the above script I get the following numbers:

./comments_bench.pl 
                 Rate   regex_assem        method1 regex_overhead      one_regex
regex_assem    2.38/s            --           -21%           -86%           -88%
method1        3.00/s           26%             --           -83%           -85%
regex_overhead 17.5/s          635%           482%             --           -13%
one_regex      20.0/s          742%           567%            15%             --

These results match up with Aaron Crane’s comment about a major speed increase. Even the test with extra overhead is multiple times faster than the method1 I had. So now I go back to this script I wrote and change it to use regular expressions with alternation. I tested these two versions I now had against a test data set of a few hundred files totaling 36 megabytes. The old script took 37 seconds to process the test data while the new version only takes 9 seconds, 25% of the baseline. Bumping up the test data to 135mb the old way takes 105 seconds, the new way 17 seconds or 16% of the baseline.

What can I say? The Perl community came through for me. I wrote an article about how I solved a problem and two people came along and gave me advice that lead to a faster solution and since speed was the goal, a better solution. Thank you.

That feature alone makes it better than most of the IDEs out there for Perl development but would we expect less? It is written in Perl by Perl developers and they have a good grasp on what information developers want. Next I wrote a little code and then opened the Regex Editor to give it a shot. I like the list of quick references on the right for things like Character classes. Setting ixmsg flags are done via check boxes at the top of the window. Inputting some test data and a simple regexp I run into a problem.

The regexp I used was \d{2,3} something simple. Now when no match is found the 'Matched text' area shows a 'No match' message in red. When the regexp matches a substring of the sample data we see all of the sample text again in the matched text area but the actual match is not highlighted. I expected it to highlight the matches in this area as a visual aid to find the matches quicker. I tried substitution and the replacements were not highlighted either.

Moving on I started looking for the must have features that all good code editors have.

• cross platform
• full unicode and utf-8 support
• visible line numbers
• current line & column number
• syntax highlighting
• brace matching
• auto indentation
• view spaces and end of line characters
• Different files as tabs
• multiple level undo/redo
• code folding
• block commenting
• new line conversion
• Perl integration (syntax check, debugging)
• text zoom
• double click word selection
• triple click line selection
• find in files
• regular expression search and replace (Only in the Replace option. The normal find does not support regexp)
• multiple instances
• generalized autocomplete
• session state preservation (only if you choose to save it)

I found almost everything I expected in Padre and the rest via plugins. Padre::Plugin::PerlTidy added perltidy support after I restarted Padre and enabled the plugin via the Plugin Manager. For Padre::Plugin::PerlCritic I followed the same routine and it by default does nothing if you do not have a minimal .perlcriticrc file. With a perlcriticrc file in hand I tried running it again and now I get output. The problem though is how the output is displayed. First off this plugin does not honor the verbose setting from the config file. No matter what I set it to the Padre output is the same. The second problem is the lack of colorized output or other indicator of the severity level of a warning message.

Next I went to Tools, Preferences to change a few options around. When I saved the changes the File menu bar disappeared, which is no good. I started up a test Windows XP environment, installed DWIM Perl 5.14.2.1 (v7) and tried the same process again and it did not mess up. So after having to restart Padre a bunch of times after making configuration changes I started using it to update a few scripts I wrote last month. Four segmentation faults later and I am done with it. I have no interest in fiddling with it to figure out the problem, I tried it out for research and now I know what it can and cannot do.

Would I recommend Padre to other developers? No.

Padre uses the Scintilla library which is where it gets the bulk of its features from. SciTE is an editor developed by the Scintilla developers that does most of what Padre does without crashing constantly, is updated more frequently and is cross platform as well. The extra features like CPAN Explorer, refactoring, and other Perl specific goodies I already have via command line applications. I started out excited to try Padre in the hopes it could make things a little faster in the development cycle. Now I just regard it as another buggy IDE. And please do not feed me a junk line about it works on my Linux distribution, I do not care. Pretend I am a normal end user for a moment. If I install a piece of software and it crashes repeatedly I simply switch to another application. I am not compelled to try and fix it or figure out what is going on, I am not invested in it as a first time user.


cpan Compress::Zlib LWP::UserAgent


Is a quick test to see what formats you can accept. 'gzip, x-gzip, deflate, x-bzip2' is the output on my system. Now lets fetch a web page and see what we get.

Looking through all the headers we can see that content-encoding is gzip for reddit. Now instead of dumping all the headers we could simply look at $response->header('content-encoding') and have what we need.

There is an interesting tidbit in the full headers of reddit:

======(  $response->{_headers}  [ 'delivery_formats.pl', line 19 ]======

    bless({

      "client-date"         => "Sat, 20 Oct 2012 18:27:48 GMT",

      "client-peer"         => "165.254.27.97:80",

      "client-response-num" => 1,

      "connection"          => "close",

      "content-encoding"    => "gzip",

      "content-length"      => 8346,

      "content-type"        => "application/json; charset=UTF-8",

      "date"                => "Sat, 20 Oct 2012 18:27:48 GMT",

      "server"              => "'; DROP TABLE servertypes; --",

      "set-cookie"          => "reddit_first=%7B%22firsttime%22%3A%20%22first%22%7D; Domain=reddit.com; expires=Thu, 31 Dec 2037 23:59:59 GMT; Path=/",

      "vary"                => "Accept-Encoding",

    }, "HTTP::Headers")

Lets see this is coming from server "'; DROP TABLE servertypes; --". A SQL injection as the server name, it makes me smile. Obviously the reddit developers have read that xkcd comic before. To protect your application against such an "attack" I would recommend reading bobby-tables.com which is a guide to preventing SQL injection.


grep -ri -f patterns files/


1125978	1	1	386s	386s	main::CORE:match (opcode)
1130211	2	1	1.09s	1.09s	main::CORE:regcomp (opcode)
41804	6	2	378ms	638ms	File::Spec::Unix::canonpath (recurses: max depth 1, inclusive time 159ms)
23226	20	7	131ms	518ms	Path::Class::Dir::stringify
250764	6	1	87.5ms	308ms	File::Spec::Unix::CORE:subst (opcode)
32515	6	2	85.6ms	435ms	File::Spec::Unix::catdir
1236	2	2	56.0ms	447ms	File::Spec::Unix::abs2rel
24462	3	2	49.0ms	49.0ms	File::Spec::Unix::catpath
5968	10	5	43.1ms	325ms	Path::Class::File::stringify
31924	10	3	42.1ms	42.1ms	Path::Class::Entity::_spec
4903	1	1	27.5ms	146ms	File::Spec::Unix::catfile
120	2	2	26.1ms	29.4ms	utf8::SWASHNEW
980	974	5	19.3ms	20.3ms	Encode::utf8::decode_xs (xsub)
6988	6	3	17.0ms	19.2ms	File::Spec::Unix::splitpath
1840	4	1	15.4ms	655ms	Path::Class::Rule::test (recurses: max depth 2, inclusive time 674ms)

1,125,978 matches in 3 megabytes of sample text files took 6 minutes and 26 seconds. This is way too slow so, let us optimize.

Divide

I started by looking at the patterns file and the majority of them are string literals which is good news. With a little preprocessing I now have all the string literals in a hash whose key is the first letter of the pattern and the value is a hash containing the patterns as keys and the values are coderefs to do whatever processing is needed. Now instead of having to run all of the literal matches every time I can test the first letter and use that to limit the matching scope. The best case scenario is 'x' which only has 1 pattern to test and the worst is 's' with over 700 patterns.

index instead of m//

Now you might be asking yourself why didn't I just use index since I have string literals? Index is designed for fast string searching with literal matches. Lets take a look at a simple example which loads 4,000 patterns and matches it against the entire text of Bram Stoker's Dracula which is a 16,248 line, 836kb file.

Is the sample program and it takes 2 minutes, 10 seconds to run. The bulk of the time is spent doing 4,000 match operations. Now lets try the same thing with index instead.

This program processes the entire Dracula text in 4.1 seconds.

The reason I didn't use index is that some of the literal patterns match as substrings of other words so I get a bunch of false positives. This does not happen with m// because I used \b to match up word boundaries. This was a precautionary measure because with that many input patterns I had to assume there would be some overlap and it turns out there is. A gentle look at simple string operations is a good read if this is new material and chapter 9 of "Mastering Algorithms with Perl" goes into depth about Boyer-Moore which Perl and grep both use.

I need tokens

This is what I had:

Which was fine but now with the need to test the first letter of each word, I need tokens. One of the advantages to making this change is I can now do text normalization on it, specifically removing duplicate tokens so less match attempts need to be made. List::MoreUtils makes this a breeze.

I read in the whole source file and strip out all the characters I do not care about. Then I split it into tokens using whitespace as the separator. Next I filter out any empty or one letter tokens. uniq from List::MoreUtils returns a list with no duplicate values. I was surprised that between List::MoreUtils and List::Util there was no function for getting the shortest or longest string in a list. Then again it is a trivial task to implement them using reduce as shown above.

Conquer

Going back to the initial 3mb of text files this updated script takes under 3 seconds to execute, that is less than 1% of the baseline time. I can now run this script 127 times in the time it took for the first version to complete one run.

More optimization

The more I used this little application on larger and larger datasets the more I wondered if I could get a little more speed out of it. I started by increasing my sample data size to 26 megabytes to simulate a more normal workload. Then I took a look at that foreach loop to see if I could speed things up. I created a simple benchmark script to compare five variations on a solution and then time them.

          Rate method5 method4 method3 method2 method1
method5  667/s      --     -0%    -34%    -40%    -41%
method4  670/s      0%      --    -34%    -40%    -41%
method3 1010/s     51%     51%      --     -9%    -11%
method2 1115/s     67%     66%     10%      --     -1%
method1 1130/s     69%     69%     12%      1%      --

Method1 turned out to be the fastest but what does this translate to in real work? Method5 takes 33 seconds to process 26mb of text while method1 only takes 22 seconds, a 33.33% reduction in execution time. For much larger datasets I am looking into doing things in parallel however the overhead of forking is greater than the amount of work that needs to be done on a per file basis.

Now I don't want to get off on a rant here ;)

I read articles like Perl 5.16 and beyond and the whole focus is new syntactic sugar and how adding a MOP to Perl which will make Moose faster. The only mention of something becoming faster is by adding all new features to Perl and that only makes certain things faster. What about the rest of us? I try to keep up the good tenets of functional programming in my applications and I have never had the need to use Moose in an application. I have literally dozens of Perl programs on my computer that I use in a day to day fashion some I wrote, some I didn't and none of them use Moose, Mouse, Moo, etc.. I want improvements to Perl that make my existing applications faster.

C was initially developed between 1969 and 1973, making it 40 years old. Java 1 was released at the beginning of 1996 making it 16 years old. Perl was first released in 1987 making it 25 years old. GCC and clang for C and the Oracle JVM get faster. If you take newer versions and test against older versions with the same code there is a trend for the newer compilers and JVM to have much better performance. The same cannot be said for Perl. Perl::Formance is an ongoing project that tracks the performance of Perl versions against different benchmark programs. The general trend is Perl performance is a plateau and gets slightly worse in some cases.

This performance and optimization stagnation Perl is in does nothing to help expand its market share and improve its image. The reality we all already know is that there is a finite amount of programmers in the world and Perl is vying for position against the constant influx of new languages as well as competing against existing languages. Some may consider the community staying the same size a good thing but my point of view on the matter is the community is either growing or dying. CPAN may seem magical to some but it is just a group of developers who choose to build libraries in and for Perl. For CPAN to stay vibrant it needs to add modules that meet the ever expanding needs of the programming world. An example of an under represented segment is graph database support. The PHP and Java community have us beat bad in those areas. Why? One of the reasons is both those languages have tools that allow them to scale far beyond the base implementation which is an important scalability concern.

I can take a PHP application that is slower than its Perl equivalent, and without changing the code compile it with HipHop and now it is faster than the Perl version. Does this work the same with B::C or B::CC? No. This is the end of my rant.

The top features I want to see in Perl:

1. A faster perl across the board
2. Better gc which would probably make applications faster
3. XS, FFI improvements to allow faster support of new libraries in the wild

Everything else for me is unimportant. People still use C because it is fast not because they are constantly adding new features to the language. On a side note I just saw a tweet from chromatic stating he has an idea of how to do better exceptions in core and stay backwards compatible.

So I am giving a data mining talk at Ohio LinuxFest 2012 and surprise, surprise there is going to be a nice helping of Perl. So I am on the internet doing research looking for some simple scrapers and collectors to mention in my talk. I always prefer to give multiple examples for any problem since programming does not have a one size fits all model. To make a long story short I found a bunch of different social media scapers. The problem I found with most of them was the same. Things like this Ruby example:

another Ruby example (the comment is from the original source):

The Python ones I found were a little more deceptive. Here is what I found on the surface:

So I see BeautifulSoup included and I am thinking that must be like HTML::Parser right? Wrong. Instead I find these:

What about HTMLParser? More of the same:

The one that probably wasted the most brain cells was this Perl one

Wow, just wow. The untold hours of work to build the above expressions and in some cases knowing that it will break no matter what.

HTML parsing with regexp is the cthulhu way and yet people still do it even though good parsers exist and have already solved this problem. HTML 4.01 was published in 2000 and HTML5 in 2008. HTML::Parser released version 2.14 in 1998 and HTML::TreeBuilder released version 0.50 in 1998 as well.

I shouldn't be surprised considering the empowerment I feel when using regular expressions to solve problems. Then we see things like this `perl -wle 'print "Prime" if (1 x shift) !~ /^1?$|^(11+?)\1+$/' [number]` which can determine if a number is prime or not. Neil already wrote up a walk through of the pattern, if your interested.

Concurrently, I am also thinking "I am not the only person who has had this problem. I bet someone already solved it on the Internet." A little searching and I had found a half dozen examples of using actual parsing libraries to work with HTML. Let me repeat the important part there 'I am not the only person who has had this problem.' This is one of those ideas that should be impressed upon programmers regularly. Tunnel vision while focused on a project happens and it only makes the final solution worse, not better. Another version of this is the 'Not Invented Here' syndrome that seems to invade programmers minds and make them think they can not only do the task better but it will be quicker for them to just rebuild it from scratch. If this happens to you take a step back and really assess the amount of time needed to write something new from scratch.

With all that said I would remind people that htmlparsing.com is an expanding community resource that can help people understand how easy it is to use a parser and not waste brain time on a regexp solution.

    my $mech = WWW::Mechanize->new();
    $mech->get('http://news.ycombinator.com/');

    foreach my $link ( $mech->links() ) {

        if ( $link->url() =~ m/perl[.]org/xms ) {

            say $1;

        }

    }

Is all it takes to get started. I would like to note that PHP and Python both have a good parsing library in html5lib.

This script generates the above message with `perl -d:NYTProf`

and this one does not.

Just switching the order of autodie and utf8::all fixed this. My mind was blown. I could not think of another time I had seen the use order do something like this. So I did some searching to see if this was known or not. Bug 70211 says it is 'use open qw( :encoding(UTF-8) :std );' followed by a use module statement that causes the issue. So my "fix" is not really a fix but there appears to be no actual problems with nytprof but I am keeping an eye on it.

Like all problems it really depends on context. Take Jon Bentley's Programming pearls: a spelling checker where he examines the problem space and the differences between a spell checker and a spelling corrector. I start by searching the keyword 'spell' across all of CPAN.

wget http://www.cpan.org/modules/01modules.index.html
ack -i spell 01modules.index.html

The above covered all 22,442 distribution names but not the sub modules names. A few metacpan searches later and I was able to compile the following list.

Direct checkers - modules that actually do the spell checking

• Lingua::Ispell A module encapsulating access to the Ispell program via IPC::Open2


• Meta::Tool::Aspell run aspell for you. Meta is a class library of about 250 classes and is abandonware.


• Text::Aspell Perl interface to the GNU Aspell library


• Text::Hunspell Perl interface to the GNU Hunspell library


• Text::Ispell A wrapper module for Ispell. The ispell cli is called via IPC::Open2.

Indirect - relies on another module to do the actual checking

• Search::Tools::SpellCheck Uses Text::Aspell to offer spelling suggestions


• Text::SpellChecker OO interface for spell-checking a block of text. Uses either Text::Aspell or Text::Hunspell

POD only checkers

• Pod::Spell::CommonMistakes Catches common typos in POD by using Pod::Spell to format the text and then comparing it against a custom wordlist from Pod::Spell::CommonMistakes::WordList. No system spell checker is required.


• Pod::Spelling Send POD to a spelling checker using either Lingua::Ispell or Text::Aspell. A test library is provided via Test::Pod::Spelling


• Test::Spelling check for spelling errors in POD files. Pod::Spell is used for parsing and an open3 call is made to either 'spell', 'aspell', 'ispell', or 'hunspell' for spell checking.

XML

• Apache::AxKit::Language::SpellCheck is an XML Text Spell Checker for the Apache AxKit. Checking is done via Text::Aspell


• xml_spellcheck is a cli application for spell checking XML files. It makes a system call to 'aspell -c' directly.

Spell checking as a test

• Dist::Zilla::Plugin::PodSpellingTests is DEPRECATED! The old name of the PodSpelling plugin


• Dist::Zilla::Plugin::SpellingCommonMistakesTests Generates a Test::Pod::Spell::CommonMistakes release test


• Test::Pod::Spelling::CommonMistakes Checks POD for common spelling mistakes using Pod::Spell::CommonMistakes.


• Dist::Zilla::Plugin::Test::PodSpelling Generates a Test::Spelling author test


• Perl::Critic::Policy::Documentation::PodSpelling Spell check the POD. Aspell is used via an open command.

Checks spelling via remote service/application

• Bing::Search::Source::Spell uses Bing to spell check text.


• Lingua::AtD Provides an OO wrapper for After the Deadline grammar and spelling service.


• Lingua::MSWordSpell Uses Microsoft Word's Spellchecker over OLE automation instead of something like ispell


• Net::Google::Spelling simple OOP-ish interface to the Google SOAP API for spelling suggestions. This appears abandoned based on last update date, number of open bugs and the fact it has more failed test reports than passes.


• WebService::KoreanSpeller A Korean spell checker

Everything else

• Gtk2::Spell Perl bindings to GtkSpell, used in concert with Gtk2::TextView.


• Lingua::Jspell Perl interface to the Jspell morphological analyzer.


• Lingua::Spelling::Alternative Use affix files generated by the ispell tools to return alternative spellings of a given word


• Pod::Spell a formatter for spell checking Pod, no actual checking capabilities built in.


• Text::SpellChecker::GUI Implements a user interface to Text::SpellChecker


• Tie::Ispell Ties a hash with an Ispell dictionary


• tkispell Perl/Tk user interface for Ispell

While many of these modules are actively developed and useful many do not fit my requirements for this project. I want to spell check any kind of utf8 encoded text and not need an Internet connection or closed source program to accomplish this task. The first two groups direct and indirect spell checkers appear to meet these requirements.

So which one to use? Lets take a look under the hood. GNU Ispell gives spelling suggestions if a word is not found in its dictionary. When searching for possible corrections to present it uses a Damerau–Levenshtein distance of 1.

GNU Aspell is an Ispell replacement that can handle utf-8 by default, has 70 supported language dictionaries, and supports using multiple dictionaries at once.

Hunspell is an advanced spell checker based on MySpell that supports both dictionaries and rules. It is currently used by LibreOffice/OpenOffice and has dictionaries for 99 languages.

It looks like Aspell and Hunspell are the front runners. I rejected Meta::Tool::Aspell because it has not been updated since 2002 and the distribution it is a part of has tons of other modules that are not needed for this problem. That leaves Text::Aspell, Text::Hunspell, Search::Tools::SpellCheck, and Text::SpellChecker left. Now before I started trying these modules out, I decided to try the command line versions of aspell and hunspell against some test data and to compare the output.

Long story short they both generate too much noise due to how they function. If a word is in the dictionary or it can be matched then everything works out. Since I am dealing with text of any kind from source code files to memos and email there is too much noise. Things like company names, places, peoples' names, animals, plants, email addresses, and technical terms can easily be flagged as incorrect.

I wrote a few sample programs to try and work around this problem. I will cover the results of my research in a future post.

01_fetch_project_list.pl builds a list of all the projects on Berlios and writes it to a file.
02_download_repos.pl takes that data file and downloads everything it can.

I broke the whole process into two scripts for a variety of reasons, mainly so I could resume the project downloads by having it premapped and then skipping anything that already existed.

Now when downloading any large number of web pages two key optimizations can be made: caching and compression. WWW::Mechanize::Cached::GZip works out great because it requests a gzip-compressed response which is fantastic for web pages and automatically caches the results for later. When downloading the projects themselves I was simply fetching archive files so WWW::Mechanize::* is just plain overkill in terms of features. LWP::UserAgent was perfect for this simple task.

I didn't need to worry about unicode since berlios uses only ASCII characters for project names. For logging the log filename needed to contain YYYY-MM-DD and automatically rotate itself. A combination of Log::Dispatch::File::Stamped and Log::Dispatch::Screen fulfilled these requirements. I also considered doing the work in parallel but decided against it. I prefer to collect data and be a nice web citizen about it instead of just slamming a server with as many requests as possible constantly.