Here is a sample invocation from the web-site:

shlomif@telaviv1:~$ sky up /home/music/Music/mp3s/Thierry\ de\ Massia/Elements\ essentiels/01\ -\ Le\ vent.ogg
X11 forwarding request failed on channel 0
sending incremental file list
01 - Le vent.ogg
     5271287 100%  147.24kB/s    0:00:34 (xfer#1, to-check=0/1)

sent 5272011 bytes  received 31 bytes  87141.19 bytes/sec
total size is 5271287  speedup is 1.00
Got URL:
http://www.shlomifish.org/Files/files/music/mp3-ogg/01%20-%20Le%20vent.ogg
shlomif@telaviv1:~$

As you can see, we just specified the file, and it divined the rsync command and gave us the resultant URL. Convenient.

---
default_site: homepage
sites:
    homepage:
        base_upload_cmd:
            - 'rsync'
            - '-a'
            - '-v'
            - '--progress'
            - '--inplace'
        dest_upload_prefix: 'hostgator:public_html/'
        dest_upload_url_prefix: 'http://www.shlomifish.org/'
        sections:
            code:
                basename_re: '\.(?:pl|pm|c|py)\z'
                target_dir: 'Files/files/code/'
            music:
                basename_re: '\.(?:mp3|ogg|wav|aac|m4a)\z'
                target_dir: 'Files/files/music/mp3-ogg/'
            video:
                basename_re: '\.(?:webm|flv|avi|mpeg|mpg|mp4|ogv)\z'
                target_dir: 'Files/files/video/'

And it uses the basename_re to figure out where to upload it later. The next features I would like to see added are an ability to specify the target directory or alternatively override the target section, and naturally - better documentation. But I'm happy with what I have so far.

CPAN Cleanups

I have been going over many of my CPAN distributions and doing various cleanups such as removing trailing whitespace (with tests based on Test-TrailingSpace), or converting the Changes file to be compliant with CPAN-Changes. It was kinda fun, though time consuming.

New Version of Freecell Solver

There's a new stable version of Freecell Solver, 3.18.0 released with some improvements including a port to JavaScript.

Black Hole Solitaire Solver

I have done a lot of work on Black Hole Solitaire Solver, and it can now also solve All-in-a-Row-Solitaire, and has some other improvements.

Binary Puzzle Solver

The Binary Puzzle Solver (written in Ruby) is now at version 0.0.3, with some bug fixes, several new moves and a convenient command line application.

I was reminded of it when working on a few new or existing distributions using Dist-Zilla. I ran into a few places where some of its plugins or features were underdocumented, and non-exemplified (in the SYNOPSIS sections or wherever), which consumed many hours of my time, sometimes made me need to read or grep the code, and made me frustrated. While I sent the maintainers some pull requests for better documenting this (and some of them were applied), I would have prefered for them to be properly documented and exemplified in the first place.

While I believe most of my CPAN distributions are in better condition, there are some aspects that are lacking, and I would like to fix them: more examples outside the SYNOPSES, and better SYNOPSES; documenting all methods of the super classes as well as the classes themselves for XML-LibXML; adding support for DTD introspection and the oft requested DOM elements annotations for it; more kwalitee; etc. and naturally also better publicity and advertising (Freecode.com (formerly known as Freshmeat.net), Reddit, blogs.perl.org, video screencasts, etc.) because people should see how cool your code is. I'm not sure if any of these things would be as blog worthy as new CPAN distributions, but it will make CPAN a bit better.

(Now that I think of it, perhaps a CPAN janitors initiative, which will submit pull requests and patches for existing distributions to improve their quality will be a good idea.)

Anyway, I am going to split this report into sections:

XML-GrammarBase

After some work (and misunderstanding of some of the Moo / MooX documentation, which resulted in some pull requests), I released XML-GrammarBase which aims provides roles and base classes for easily writing wrappers around XML schema formats and XML translation languages. So far only RELAXNG and XSLT are supported. Here is the synopsis from the docs:

package XML::Grammar::MyGrammar::ToOtherStuff;

use MooX 'late';

use XML::GrammarBase::Role::RelaxNG;
use XML::GrammarBase::Role::XSLT;

with ('XML::GrammarBase::Role::RelaxNG');
with XSLT(output_format => 'html');
with XSLT(output_format => 'docbook');

has '+module_base' => (default => 'XML-Grammar-MyGrammar');
has '+rng_schema_basename' => (default => 'my-grammar.rng');

has '+to_html_xslt_transform_basename' => (default => 'mygrammar-xml-to-html.xslt');
has '+to_docbook_xslt_transform_basename' => (default => 'mygrammar-xml-to-docbook.xslt');

package main;

my $xslt = XML::Grammar::MyGrammar::ToOtherStuff->new(
    data_dir => "/path/to/data-dir",
);

# Throws an exception on failure.
my $as_html = $xslt->perform_xslt_translation(
    {
        output_format => 'html'
        source => {file => $input_filename, },
        output => "string",
    }
);

perform_xslt_translation supports myriad options and formats, and you get it all for free.

I've also been contemplating something like App::XML::GrammarBase or XML::GrammarBase::App that will allow for easy registry of such formats from the command line and for processing them.

At first, I believed that Moo was "almost-completely-unlike-Moose", but «use MooX 'late';» makes it much more Moose-like and I enjoy it, and it is now recommended over Any::Moose or maybe even plain Mouse. Mouse is very fast as well and has a far smaller startup time than Moose, but it does not interoperate very well with pure-Moose classes, which Moo does well.

HTML-Widgets-NavMenu-ToJSON

As I was working on my homepage, I noticed that after I placed all the sub-navigation menus inside the main navigation menu, it increased the size of the navigation menu considerably, having it consume over 40,000 bytes. I decided to put only a subset of it as HTML, and load and populate the rest using an AJAX call to a common and static .json file. So I needed a way to convert the input of HTML-Widgets-NavMenu

So enter HTML::Widgets::NavMenu::ToJSON, which serialises a navigation menu structure as JSON. I also implemented a persistence layer for it, currently with only one backend - a YAML file, so the IDs which jqTree require would be, well, persisted. This is a bit of an architecture astronautics, but I ended up with something concrete at the end so it's under control.

Here is the synopsis:

use HTML::Widgets::NavMenu::ToJSON;
use HTML::Widgets::NavMenu::ToJSON::Data_Persistence::YAML;

my $persistence =
    HTML::Widgets::NavMenu::ToJSON::Data_Persistence::YAML->new(
        {
            filename => '/path/to/persistence_data.yaml',
        }
    );

my $obj = HTML::Widgets::NavMenu::ToJSON->new(
    {
        data_persistence_store => $persistence,
        # The one given as input to HTML::Widgets::NavMenu
        tree_contents => $tree_contents,
    }
);

use IO::All;

io->file('output.json')->println(
    $obj->output_as_json(
        {
            %args
        }
    )
);

It could be made a bit simpler using an abstraction, but I feel it's not too bad. You can write custom HTML::Widgets::NavMenu::ToJSON::Data_Persistence child classes by implementing the load() and save() methods - I did not document the process yet, but you can base your efforts on HTML::Widgets::NavMenu::ToJSON::Data_Persistence::YAML.

Dist-Zilla-Plugin-TestRun

Since I'm used to running ./Build runtest instead of ./Build test for my distributions, I sought a way of doing the same using Dist-Zilla. It turned out there wasn't an easy way, and Dist-Zilla-Plugin-TestRun is the best I could do. I hope to write a patch to make the test command more configurable.

Test-XML-Ordered

Test::XML::Ordered makes a non-"semantic" diff of XML files and tells you if they are equivalent (up to differences of whitespace and namespace prefixes). It grew out of my frustration with XML::SemanticDiff, upon which Test::XML is based, and which I maintain. A quick CPAN search revealed other modules for testing XML in a similar manner, but since I have already written my code, I decided to release it.

One tip is that with the current API, one should use something like:

my @common = (validation => 0, load_ext_dtd => 0, no_network => 1);

is_xml_ordered (
    [ string => normalize_xml($results_buffer), @common, ],
    [ location => "./t/data/xhtml-results/$fn_base.xhtml", @common, ],
    "Testing for Good XSLTing of '$fn_base'",
);

In order to test XHTML documents for equivalency.

Vered-XML

I thought that Vered-XML (currently only available in the Perl Beginners' Site's repository was probably the most ad-hoc and tailored to match XML grammar I have ever created, and was used to convert the markup of the "Perl Elements to Avoid" page from hacky Website META Language markup to XML, so it will be easier to translate to different (human) languages and also somewhat easier to maintain and render. Currently it renders to DocBook/XML, which allows translation to many other formats.

So why did I call it Vered? Since it was so ad-hoc, and I didn't know what else it would be useful for, I wanted to give it an artsy name, and considered using a random Hebrew feminine name. Then I thought that "a rose by any other name would smell as sweer", and thought of calling it Rose, but translated to Hebrew it was "Vered", which is also a Hebrew feminine name. I still have to tie up some loose ends, but expect XML-Grammar-Vered at a CPAN mirror near you Real Soon Now™.

Vered-XML supports a subset of XHTML, as well as some specialised tags such as <pdoc /> (perldoc), <pdoc_f /> (perldoc -f), <cpan_self_mod /> (CPAN self link to module), etc. and contains a RELAX NG schema and an xsltproc/libxslt2-compatible XSLT 1.0 stylesheet. Here is an excerpt from the "Perl Elements to Avoid" source XML:

<item xml:id="slurp">
<info>
<title>Slurping a file (i.e: Reading it all into memory)</title>
</info>
<p>
One can see several bad ways to read a file into memory in Perl. Among them
are:
</p>
<code_blk syntax="perl">
# Not portable and suffers from possible
# shell code injection.
my $contents = `cat $filename`;
# Wasteful of CPU and memory:
my $contents = join("", &lt;$fh&gt;);
# Even more so:
my $contents = '';
while (my $line = &lt;$fh&gt;)
{
    $contents .= $line;
}
</code_blk>
<p>
You should avoid them all. Instead the proper way to read an entire file
into a long string is to either use CPAN distributions for that such as
<cpan_self_dist d="File-Slurp" /> or
<cpan_self_dist d="IO-All" />, or alternatively
write down the following function and use it:
</p>
<code_blk syntax="perl">
sub _slurp
{
    my $filename = shift;
    open my $in, '&lt;', $filename
        or die "Cannot open '$filename' for slurping - $!";
    local $/;
    my $contents = &lt;$in&gt;;
    close($in);
    return $contents;
}
</code_blk>
</item>

I'm not sure if Vered-XML will have much utility outside that use case, but I think that putting it on CPAN won't hurt much.

Improving perlresume.org

I filed a few issues and sent some pull requests for perlresume.org for some problems I ran into, and vti fixed or applied them promptly. Thanks!

Beginners' Sites

I invested more work into the Perl Beginners' site and also, after some encouragement from the good people on #vim began work on the Vim Beginners's site, which I publicised on my tech blog, on Reddit, and on some mailing lists. While in the past I used my own static site generator called "Latemp" based on Website Meta Language for sites like that, I used Jekyll this time, at least for the first prototype. I did so in order to evaluate it for use in a different site (The Israeli Linux Portal which is currently implemented using an awkward custom PHP-based system, that mostly just reads stuff out of XML data files and outputs a constant static HTML as XSLT.). While at first I was impressed with Jekyll, I then realised that it was quite opaque, often required many plugins to get some basic functionality and that one plugin I looked at lacked examples and was otherwise under documented.

There is a list of many more static site generators / offline content-management systems around (which does not include my own Latemp), and it may be easier to roll up something on my own based on Template Toolkit or its ttree. All this just confirms my (and Su-Shee's) suspicion that every self-respecting web programmer has written at least one.

Freecell Solver for JavaScript - ☺

I have been distracted for many hours trying to port Freecell Solver to JavaScript using the very cool Emscripten LLVM bitcode to JavaScript compiler. The mostly final result could still use some work, and is still under optimised (in large part due to what appear to be an Emscripten bug) but it's working.

The Emscripten wiki contains links to many, much more impressive, demos including a JavaScript version of the open-source graphical game "Me and My Shadow".

Two Anecdotes from ##programming

I enjoy chatting on ##programming (on Freenode) and along with all the rants about how much every popular programming langauge in existence sucks, and trying to help various students with their often badly indented and badly formatted homework code, there are often some insightful discussions by some smart people. I'd like to share two Perl-related anecdotes from there.

The first one was that someone there who has taken studying programming, before starting to study in university, seriously, and also read the Camel book and other books and resources about Perl, asked me to look at my code to see how readable it is. I referred him to the source code of HTML::Widgets::NavMenu, which will being an early code of mine (and still not fully modernised), was written in a modular manner, and he said he found it to be well-factored and yet very readable . He noted that he suspected that Perl would be less believed to "hard to understand" or "unreadable", if it sandardised on using my preferred style - Allman style - instead of its default style with the opening braces on the same line as the opening clause (e.g: if ( COND ) {).

The other anecdote was that someone else there, who is working in Visual Basic and Visual Basic for Applications for a living (for lack of good alternative for software development job prospects in his homeland), told us that he found the CPAN documentation to usually be superior than the MSDN (Microsoft Developers Network) one. I found it a little strange, because I recall finding the MSDN documentation to give detailed examples for almost every subroutine and a page of prose and explanation for each one, and reportedly large parts of it were also translated into some non-English languages by Microsoft. Then I showed him the "SYNOPSIS" sections of some of the .pm files in CPAN and he was even more impressed and said it was useful (because you can copy+paste it and tweak it), but naturally commented that not all .pm files had those.

Conclusion

I currently stand at 87 CPAN distributions, some of which a bit lacking, and I still have some in the pipe. One of them will require quite a lot of cleanup work to do, but it's not a high priority. I also have to juggle some other endeavours, and also recently started contracting for an Internet friend as a Drupal developer (and in the future possibly one for Moodle as well), which may consume even more of my time (but on the bright side pay a little money). It's no secret that I am not a big fan of PHP, but Drupal seems impressive so far, and hides away a lot of PHP's ugliness. It took me some time to get acquainted with it, so I hope now things will be smoother and easier to do. It seems that there's quite a bit of demand for converting sites into Drupal, setting up new sites, or maintaining them in Israel (and possibly elsewhere as well) so it is probably a useful skill to have under my belt.

Hope you enjoyed this hacktivity log, and see you next time. Byeeeee!

The first upload is SDLx-Betweener, which allows for creating high-performance and smooth animations in SDL by making use of Perl/XS. Almost all of the coding (and a related YAPC::Israel talk) was done by Ran Eilam, who is a very cool guy, a good friend and a former boss of mine, and I've just done (with his permission) some last-minute cleanups and did the initial upload to CPAN. So I kinda feel like the frog that sat on top of the elephant who crossed the river and said "We did it!".

However, I needed SDLx::Betweener for something I've been intending to work on, and wanted it on CPAN and packaged in Mageia Linux, so I did that. If you run into any problems with it, drop me a note or file a bug report, and hopefully I can resolve these problems (possibly with some help from Ran).

The next distribution is original, but its story is more complicated. It all started when I decided to resume work on Qantor, which aims to eventually be a modern and saner alternative to TeX/LaTeX and Troff (although it is still extremely far from that now). I noticed that the parsing was still done by Regexp::Grammars, and so decided to convert it into Parser::MGC. Parser::MGC worked eventually and I was impressed from the straightforward way to do stuff with it, but it was time consuming to get there (like most other parser generators I tried, only a bit better), and involved writing some Moose 'around' code to debug the method calls, and also sometimes delving into the code. Parser::MGC is still not perfect (and some of the code I read there had gems like die $e if $committed or not eval { $e->isa( "Parser::MGC::Failure" ) }; which I found hard to parse, and made me want to drown a kitten), but I think it sucks less and is more transparent than other parser generators I tried.

Anyway, after I got it working, I noticed that the Test::XML::Ordered code (then still unreleased), generated a some memory problems and I suspected either XML::LibXML or libxml2 to be the culprit. This involved a long investigation process with valgrind, gdb, perl -d and other tools which culminated in a very small change and more lines of test code. This was released in XML::LibXML. After that I received a report that the tests got stuck, but that turned out to be due to external loading of DTDs and was easily fixed. There's another report for a problem with perl-5.8.8 (ouch!) on Red Hat Enterprise Linux (ouch again!), but I'm not too motivated to do a lot about it (see this fortune cookie).

Well, after I fixed the problem with XML::LibXML, I decided to release Test::XML::Ordered. I'm well aware of Test::XML, but found it hard to rely on it, because it uses XML::SemanticDiff (the latter I adopted) and which attempts to rearrange the order of the nodes when it sees fit (so <ul><li>One</li><li>Two</li></ul> and <ul><li>Two</li><li>One</li></ul> may be considered the same), which was not what I want and not what other people who contacted me about XML::SemanticDiff wanted.

After Test::XML::Ordered, I decided it would be a good idea to finally work on the release of XML::GrammarBase which aims to provide "base classes and roles" for facilitating creating XML validators and processors. Work on it was pretty straightforward after I had Test::XML::Ordered available, but required some refactoring the classes from using Any::Moose to using Moo per advice of the people on #moose, and I have yet to convert the XSLT role into a parameterised role.

I have some other code I'd like to release on CPAN after all that.

Another related hacktivity was a set of patches to the core Vim that I've written to improve support for DocBook 5. Since I wanted to maintain compatibility with DocBook 4 documents (because DocBook 4 is still popular), this involved writing a Perl, Bash and Python Amalgam to generate the Vim code in question with a list of common, DocBook 4-only and DocBook 5-only tags. You can find it in its BitBucket repository. It is very hacky, but it works. Using the Python code gave me an idea for a missing XML::LibXML feature - DTD introspection - which I'd like to implement in the future.

Well, that's all for now - just wanted to get it out of my system. Enjoy the holidays, and Cheers!

I have already made some progress with the grant (and this post is part of it), and sent a log for Alan Haggai to post as a grant update.

That put aside, I'd now like to publicise some of the new distributions I have uploaded to CPAN, as I had some ideas for new ones, and am also contemplating making enough uploads to have 100 CPAN distributions or more:

1. WWW-LinkChecker-Internal is a distribution wrapping a script that checks for internal broken links. I released the first three versions of it in a day, and it uses WWW::Mechanize. The reason why I wrote it was that the WDG HTML validator reported some broken links in one of my sites, but not their origins, so I had to trace them in a different way, and ended up writing that.

2. Inline-Ruby is an Inline:: distribution I adopted that allows to embed Ruby code inside Perl programs. I applied some fixes for it (some of which were not released to CPAN yet and some are ugly workarounds), and am planning to maintain it further down the road.

3. Test-TrailingSpace is a Test:: namespace distribution for verifying that your code does not contain trailing space. Death to trailing space!

A final anecdote is that when working on one of my CPAN modules I found the b [filename]:[line_num] feature that I implemented in the perl debugger to be of great help, and can be content that I'm eating my own dogfood.

Cheers for now.

However, I was a little confused about using magpie, so I'd like to share my findings here:

1. In order to import, upload and submit a new CPAN package into Mageia, along with all of its dependencies, one should not use magpie, but rather cpan2pkg. Its use is very simple: make rpm and urpmi sudoable, and type cpan2pkg Package::Name from the command-line inside an X terminal. This will start a Tk window where one can monitor the progress of preparing new RPM packages and it has an entry box to create more packages (which saves time on re-initialising CPAN.pm or CPANPLUS.pm).

2. In order to upgrade a package, one can type eval $( magpie co -s perl-[PACKAGE_NAME] ) and then magpie update. magpie requires minicpan to be installed and updated.

3. In order to install packages, one can do sudo urpmi 'perl(Package::Name)'. My Module-Format module facilitates the translation from other notations for writing modules:

   up()
   {
       sudo urpmi --auto $(perlmf as_rpm_colon "$@")
   }