Max Maischein

Trying to hide from the Cloud

2012-09-30T18:21:06Z

I'm trying to get Plync to work with my Nexus 7, mainly because I want non-Google calendar synchronization between my mobile, my desktop and this shiny toy. Authentication works, but the Nexus 7 does not want to list the available folders at all and does not attempt to synchronize the Calendar folder.

To further debug this, having a good+free (or at least, available) ActiveSync server that I could use to debug the network traffic against would be very convenient. $work does not use ActiveSync, so it won't be much use there...

Twitter Bootstrap templates for Dancer Applications

2012-05-08T15:41:44Z

I'm writing an internal application using Dancer. To give it a "current" look, I'm using the Twitter Bootstrap CSS framework. A side effect is, that I wrote Dancer::Layout::Bootstrap, a set of Template Toolkit templates that import the Bootstrap layout into Dancer. I hope that this will evolve into a way to add external data files and maybe even layout templates to scaffolding frameworks such as Dancer.

Pretty Pictures

The demo application has a tiny bit of logic to demonstrate the effect of showing "flash messages" with Bootstrap, the green bar you see in the below image:

The issue why I haven't released this onto CPAN yet is that I'm not aware of a better way to distribute ephemeral projects like templates and Javascript via the CPAN toolchain than to package them into a module / application. Maybe Dancer will come up with an API to load data from modules or to create/update templates from modules, or somebody will point me to an appropriate method to do such things in the comments.

How many faces do you count?

2012-02-27T21:02:22Z

How many faces do you count in this image?

Perl counts 44:

(3068,1983): 25x25 @ -2.30285501480103
(4442,2028): 25x25 @ -3.287189245224
(2015,2071): 26x26 @ -3.70826029777527
(3458,2041): 31x31 @ 0.825069129467011
(3849,2063): 28x28 @ -2.27478647232056
(3328,2021): 33x33 @ -0.250598877668381
(3413,2024): 34x34 @ 0.513785362243652
(4188,2039): 31x31 @ -0.453457891941071
(2256,2058): 30x30 @ 0.173833101987839
(2809,2110): 31x31 @ -0.524535775184631
(1834,2168): 32x32 @ -1.64895510673523
(2180,2128): 34x34 @ 3.05323696136475
(3222,2446): 33x33 @ -3.49713373184204
(4663,2134): 37x37 @ 3.50697612762451
(3082,2155): 37x37 @ 2.55681991577148
(1651,2125): 36x36 @ 0.686093211174011
(3596,2133): 32x32 @ -2.47773456573486
(1887,2204): 38x38 @ 2.26138424873352
(3733,1876): 34x34 @ -2.05330586433411
(4907,2190): 42x42 @ 6.07879543304443
(4296,2092): 39x39 @ -3.16300702095032
(3188,2126): 39x39 @ -0.357136130332947
(3859,2140): 42x42 @ -0.0315352529287338
(4477,2149): 39x39 @ 3.46311140060425
(3946,2223): 40x40 @ -0.409443378448486
(906,2228): 40x40 @ -0.587313830852509
(3761,2258): 49x49 @ 5.01696395874023
(4978,2280): 46x46 @ -0.811646401882172
(1819,2335): 51x51 @ 6.60061264038086
(4071,2331): 49x49 @ 1.23171555995941
(3430,2230): 46x46 @ -0.422206252813339
(4569,2542): 52x52 @ -3.66873860359192
(5390,2236): 57x57 @ 2.28667879104614
(3980,2381): 54x54 @ -2.23162841796875
(4834,2254): 60x60 @ 0.909286737442017
(2645,2460): 59x59 @ -1.53757762908936
(3394,2470): 65x65 @ 1.81137716770172
(275,2303): 61x61 @ -3.08972954750061
(3159,2462): 80x80 @ -0.73163914680481
(4238,2356): 64x64 @ -1.11184680461884
(4067,2493): 73x73 @ 0.357052862644196
(5187,2235): 91x91 @ -3.84829616546631
(3394,1436): 243x243 @ -0.155338764190674
(189,1099): 915x915 @ -0.393947660923004

In theory, the last number should be some confidence value, but so far, I haven't made much sense of its range.

I hope to get the code packaged and released as Image::CCV before or on the German Perl Workshop 2012 in Erlangen.

How I roll - choosing the epigraph for 5.15.8

2012-02-21T18:52:07Z

Thinking about the epigraph of my Perl release started about 10 days before the release. Certainly, I had been unconsciously mulling back and forth about killer quotes from books or other media for longer than that. About 10 days ago, I made the short list of two books that could give interesting quotes.

One was "Friday" by R.A. Heinlein. I've always liked the book, and it opens with the introduction of the protagonist as she kills a pursuer and stuffs him into a cabinet, reacting only on a hunch. Heinleins way of laconic writing should have made for an interesting quote from that scene.

The second book was "The Manual" by the DJ combo The KLF. It is an unromantic look at the music business and follows a Faustian theme. It purports to be a manual for achieving a Number One in the UK in the late 1980s. It taught me interesting concepts, like for example the idea of diving head-first into a venture and the importance of the bass line in pop music. While following the manual certainly was a way to get a Number One hit in the UK, it also makes clear the other half of the faustian pact - getting the Number One is the only thing you will get, immediately in its two opening paragraphs:

Be ready to ride the big dipper of the mixed metaphor. Be ready to dip your hands in the lucky bag of life, gather the storm clouds of fantasy and anoint your own genius. Because it is only by following the clear and concise instructions contained in this book that you can realise your childish fantasies of having a Number One hit single in the official U.K. Top 40 thus guaranteeing you a place forever in the sacred annals of Pop History.

Other than achieving a Number One hit single we offer you nothing else. There will be no endless wealth. Fame will flicker and fade and sex will still be a problem. What was once yours for a few days will now enter the public domain.

Originally I had thought to use these two paragraphs as the epigraph. Especially the prophecy of short-lived fame seemed to ring well with being the release manager of the latest development release of Perl until next month. But once I had started re-reading the book, the chapter on "Chorus And Title" is about how these are the central piece of the song. The authors state that the chorus is the main emotional pull the prospective Number One will need. And as the "Doctor Who" chorus reminds at least me of soccer fans chanting in a stadium, this chorus and the following paragraph explaining the deeper relevance behind the lyrics struck a chord. The segue of the text into showing examples of real masters of producing number one hits serially, and the culmination in the Rick Astley quote sealed the deal for me. Who could resist citing a meme before it was cool?

After rereading the book, it is interesting how much the music business has changed in the sense of publicity. It is also interesting how aware the authors were at this point in time that there was a change coming in the sense of accessibility of the means of music production. Of course, much of the advice about studios has been obsoleted by the technical advances of the past 20 years. The music market is not as centralized anymore, and channels like MTV and later Youtube have eroded the influence of Top of the Pops, so marketing strategies today would have to be different. Also, the music distribution does not happen in stores much anymore.

Your Perl-driven cyborgs can soon recognize objects

2012-02-13T20:04:22Z

I've made some progress in getting ccv to compile under Windows. Now I can start with opening up the API and writing some XS for it. Soon, the below image can be created completely from within Perl instead of using a shell call out to the executable to find "interesting" points in the object and the scene and match them together.

The SIFT algorithm seems to be quite a useful tool to do feature recognition without any prior training. I hope to (ab)use this to recognize objects and how they move around in movies. Also, I imagine this can be used to automatically stitch images together.

A real fork() call could be possible on Windows (XP onwards)

2011-10-11T18:12:25Z

Prompted by BrowserUk in his post on Perlmonks on Win32, fork and XS globals, I looked into the PAGE_WRITECOPY for CreateFileMapping, and it seems that, in fact, this function, together with a bit of work could be made to implement a (data-)fork call on Windows, with real sharing of identical pages.

There is this sample program on the MSDN, which outlines how to use a mapped view of a file for shared memory IPC, and by using the PAGE_WRITECOPY flag instead of FILE_MAP_ALL_ACCESS, we should get COW semantics for the shared area.

The part for implementing fork() would seem to just be:

(and that is the hard part) Determine the range of data to be shared with the child
CreateFileMapping in the parent process to share the area from 0.
MapViewOfFile
Launch the child process
In the child process, call OpenFileMapping() to connect to the data area from the parent process.
Fix up any cached values that need changing after a fork(), for example $$

Work progress on WWW::Mechanize::NodeJs

2011-10-03T19:11:57Z

Lots of people use WWW::Mechanize::Firefox. This makes me happy.

Some of these people would like to do away with needing Firefox to be running on the machine doing the automation. To look in that direction, and to gain some familiarity with nodejs, I started porting the proxy object backend MozRepl::RemoteObject to nodejs. I've uploaded the work in progress to Github as NodeJs::RemoteObject. There is a lot of copied and pasted code between the two ::RemoteObject modules, and likely, this will beget a third, shared incarnation of (Javascript) proxy object implementations.

The main thing that's still needed is to actually write a web "browser" implementing just enough to run most web applications, for nodejs. I think there already is something called "jsdom", which claims to be just enough of a browser to make this work.

Discontinuing support for Firefox 3.0

2011-07-25T19:06:22Z

A heads up and last call for people using WWW::Mechanize::Firefox with Firefox 3.0.

I plan on discontinuing support for Firefox 3.0 due to moving to the native JSON encoder that exists starting with Firefox 3.5.

If you are a Firefox 3.0 user (for compatibility testing?), and really, /really/ need the support, please talk to me so that we can work on a solution that keeps the code small and maintainable for me. If nobody speaks up, I'll interpret that as nobody using Firefox 3.0 anymore, which is fine by me as well.

In other news, I now got several "portable" Firefox instances installed and modified the test files to test against all installed instances. This now tests Firefox versions 3.5, 4.0, 5.0 and 6.0 Beta.

Display your data - Random::PoissonDisc

2011-06-04T13:21:54Z

I read this nice article on map generation and naturally want to write something like it, but in Perl. For the first step, I want to distribute points nicely across the plane, by using the Poisson Disk Sampling method outlined in the article. Implementing the algorithm was straightforward, but how would I know that the output data was correct in the sense of reaching my goal of uniform but randomly distributed points across the plane?

The test program of Random::PoissonDisc conveniently outputs the generated points to STDOUT. For visually inspecting the data, I can then pipe it into App::ffeedflotr which displays a nice point cloud. The first output of Random::PoissonDisc looked like this:

perl -w bin/random-poissondisc.pl -r 10 | ffeedflotr.pl --type=scatter

Clearly, this picture shows that something was wrong with the first attempt of creating the random data. The grid is 20x20 units, and with a minimum distance of 10 between points, there should be at most 2 points per grid. My error was in the code for sample rejection. The code only checked the immediate cache, and not the neighbouring cache positions. After fixing that part of the code, I'm not certain whether the code is correct. But its output looks quite satisfying:

The module itself will soonish hit CPAN. I see a lot of improvement opportunities for the module. It could support incremental point generation. It could actually use a vector library like PDL instead of hand-rolling most of the vector manipulation code. It could use some tests beyond basic functionality tests.

But for the moment, I'm very happy with the fact that the code produces results I can use in creating random maps. And also I'm very happy with the fact that I have nice visualisation tools that enable me to visually verify the quality of datasets.

You can solve any problem by adding another layer of indirection

2011-04-25T11:59:48Z

During the Easter weekend, I've started to write code supporting Firefox 4 in WWW::Mechanize::Firefox. So far, only some ugly parts of the restructuring of Firefox have broken WWW::Mechanize::Firefox. As my main application for it still runs Firefox 3.x, I have to support both versions, and especially not break any functionality for 3.x. So far, I could put most of the differences between Firefox 3.x and Firefox 4 into two submodules, WWW::Mechanize::Firefox::API35 respectively WWW::Mechanize::Firefox::API40. The constructor of WWW::Mechanize::Firefox does return either of these subclasses, depending on the version. This proves that you can't solve the problem of too many indirections by adding another layer.

As I have no real use case for Firefox 4 yet, I'm not sure whether there are more differences in when (and which) events fire during page load etc.. There is one nice glimmer of improvement for Firefox 4. I've found a Selenium test file that overrides many of the modal dialogs, so that I can now maybe implement ->proxy() for WWW::Mechanize::Firefox and suppress modal dialogs from blocking the whole application.

If you have Firefox 3.x and automate a critical application, don't blindly upgrade Firefox on your main machine. Firefox 4 and Firefox 3 will not peacefully coexist if you don't take preventive measures.

If you want to do a test drive of the upcoming version, take a look at the Firefox4 branch of the Github repository.

I love it when a plan comes together

2011-04-11T19:43:00Z

Barbie blogged about the 11 million test reports mark, and gave a link to a dataset.

I had been sitting on my data "visualization" idea App::ffeedflotr for some time already, and had just started writing and publishing it on github. That dataset pushed me to write a short tutorial / showcase of the application.

The screenshots were produced by WWW::Mechanize::Firefox, the charts produced by the flot library, and the Javascript manipulation also came through MozRepl::RemoteObject. It's nice when your tools conspire to surprise you in a good way!

App-ffeedflotr - plot using Firefox and flot

2011-04-10T17:47:55Z

Sometimes, I want a pretty plot of data. Excel is not always the tool of choice,
especially when the data to plot is something like running progress of a (Perl) script I wrote,
or the output of

while /bin/true done; ls $file ; sleep 1; done

Inspired by feedGnuplot , I wrote ffeedflotr.pl, which takes data and plots it in Firefox. So far, it is not really configurable. But as I have the API blueprint of `feedGnuplot`, I can easily/conveniently adapt it to the featureset.

One nasty thing I found is that Firefox does not like to execute Javascript that comes from `data:` URIs. I suspect this is due to security concerns, but it means that I have to find a sneakier way to load custom HTML into Firefox, preferrably avoiding a tempfile. Things that I've tried so far are named pipes - Firefox does not like reading from these either, likely due to the same or similar security concerns.

Downwards compatibility, and a surprising find

2011-03-17T21:21:57Z

I've just released a new version of WWW::Mechanize::Firefox, appropriating a new function of WWW::Mechanize, ->click_button. I used that occassion to also change the XPath queries sent by W:M:F to Firefox to be somewhat slimmer by changing queries for the element type from a global "OR" to a local test for the node name:

The old query to look for a

Patching XS modules

2011-02-25T22:33:48Z

Today, I found out about a new release of Image::Thumbnail , which has support for Image::Epeg , a library coming from the Enlightenment desktop manager. Likely, epeg is quite fast, but it didn't build and test right under Windows. Conveniently though, Tokuhiro Matsuno maintains the module on github, so it was just a matter of forking and cloning his repository, and then trying to find out what made it break.

The breakage itself was three parts:

A build failure where the symbols were not exported. It seems the epeg library wants -DBUILDING_DLL , which I supplied as a cc_optimize_flag through Module::Install, because I couldn't find a better way.
Some test failures because binmode() was not used with the test image files. Easily fixed.

These two tests made it into v0.11 , released about 30 minutes after I told tokuhirom about the patches.

The remaining problem was that a function call crashed Perl with

Free to wrong pool ... at ...

This was due to a call to free(pOut) to release a memory block that the epeg library allocated for us. Unfortunately, the library uses malloc and free, and Perl redefines the two, so I had to find a way to make the XS module also call the original free() and not the redefined free(). Tye McQueen pointed out a working hack by making the compiler forget the redefinition:

#undef free

That undefinition would not have worked if there were other places below that line that still needed access to the redefined free(), but luckily that was not the case. The only snag was my limited knowledge of C preprocessor specifics, as I had some whitespace before the #undef and it was not picked up nor flagged as error. But after some trials, I also figured that out, and the pull request went out.

Cascading config parsing

2011-02-22T21:09:44Z

Prompted by some comments about how to handle RAW files from cameras, I revisited App::imagestream , the program I use to automatically publish all images I touch to a gallery page on my website.

An experiment that I run with this program (and with App::fritzgrowl as well) is to specify the configuration information via POD. There is code in Config::Spec::FromPod and Config::Cascade that takes POD and turns it into a hash of hashes containing the configuration model. Then there is more code in App::ImageStream::Config to turn this model into a DSL for a config file, a parser for Getopt::Long or simply the defaults. Config::Cascade then fills out the values, starting with the most specific values coming from the command line, the less specific values coming from the config file and the least specific values coming from the application defaults, all driven by the documentation.

=head2 C<< output DIR >>

=for config
    repeat => 1,

Output directory

Specifies the output directory into which the output will be written.

Example:

    output 'C:/ImageStream/';

May appear only once.

So far, the experiment has been successful in the sense that specifying the config items via POD forced me to document each configuration item immediately. Writing the DSL and Getopt evaluator was fun, and having them data-driven will make it easy to also inspect %ENV for an intermediate stage of configuration between command line and configuration file. Having the first line be a really short description and always listing an example also makes me think more about the values and how to best describe them.

The bad thing is of course that internationalization / localization will be nasty when specifying program data through program configuration. If you ruin the pod, the program won't work anymore, as it can't read its configuration anymore. This is currently of small concern to me as I don't plan on translating its documentation. But if some other well-meanung and unsuspecting individual translates the documentation, that'll be fun breakage at a distance.

Also bad is that my way of specifying out-of-band information implies embedding Perl again via =for blocks. This is a nasty-but-convenient hack.

In the long run, Config::Spec::FromPod should produce something suitable for consumption by Config::Model, so that I can reuse the GUI editors and the schema provided by it. But maybe I'll realize sooner that the experiment is doomed to fail, or I abandon the applications using it.

I haven't released the modules onto CPAN for two reasons. Firstly, because I think that while interesting, parsing POD for code is a debatable design choice. Second, the API of Config::Cascade isn't that great yet, and it's missing the provider that fetches the values from %ENV, and lots of use and documentation. I've only used the module in two applications, and in my experience, if I haven't used something at least three times, the API will likely be crap.