programming Archives

Trying to hide from the Cloud

I'm trying to get Plync to work with my Nexus 7, mainly because I want non-Google calendar synchronization between my mobile, my desktop and this shiny toy. Authentication works, but the Nexus 7 does not want to list the available folders at all and does not attempt to synchronize the Calendar folder.

To further debug this, having a good+free (or at least, available) ActiveSync server that I could use to debug the network traffic against would be very convenient. $work does not use ActiveSync, so it won't be much use there...

How many faces do you count?

How many faces do you count in this image?

image-ccv-face-mass.png="text-align: center; displa…

Work progress on WWW::Mechanize::NodeJs

Lots of people use WWW::Mechanize::Firefox. This makes me happy.

Some of these people would like to do away with needing Firefox to be running on the machine doing the automation. To look in that direction, and to gain some familiarity with nodejs, I started porting the proxy object backend MozRepl::RemoteObject to nodejs. I've uploaded the work in progress to Github as NodeJs::RemoteObject. There is a lot of copied and pasted code between the two ::RemoteObject modules, and likely, this will beget a third, shared incarnation of (Javascript) proxy object implementations.

The main thing that's still needed is to actually write a web "browser" implementing just enough to run most web applications, for nodejs. I think there already is something called "jsdom", which claims to be just enough of a browser to make this work.

Discontinuing support for Firefox 3.0

A heads up and last call for people using WWW::Mechanize::Firefox with Firefox 3.0.

I plan on discontinuing support for Firefox 3.0 due to moving to the native JSON encoder that exists starting with Firefox 3.5.

If you are a Firefox 3.0 user (for compatibility testing?), and really, /really/ need the support, please talk to me so that we can work on a solution that keeps the code small and maintainable for me. If nobody speaks up, I'll interpret that as nobody using Firefox 3.0 anymore, which is fine by me as well.

In other news, I now got several "portable" Firefox instances installed and modified the test files to test against all installed instances. This now tests Firefox versions 3.5, 4.0, 5.0 and 6.0 Beta.

Display your data - Random::PoissonDisc

I read this nice article on map generation and naturally want to write something like it, but in Perl. For the first step, I want to distribute points nicely across the plane, by using the Poisson Disk Sampling method outlined in the article. Implementing the algorithm was straightforward, but how would I know that the output data was correct in the sense of reaching my goal of uniform but randomly distributed points across the plane?

The test program of Random::PoissonDisc conveniently outputs the generated points to STDOUT. For visually inspecting the data, I can then pipe it into App::ffeedflotr which displays a nice point cloud. The first output of Random::PoissonDisc looked like this:

perl -w bin/random-poissondisc.pl -r 10 | ffeedflotr.pl --type=scatter

Image of wrong distribution

Clearly, this picture shows that something was wrong with the first attempt of creating the random data. The grid is 20x20 units, and with a minimum distance of 10 between points, there should be at most 2 points per grid. My error was in the code for sample rejection. The code only checked the immediate cache, and not the neighbouring cache positions. After fixing that part of the code, I'm not certain whether the code is correct. But its output looks quite satisfying:

Better distribution

The module itself will soonish hit CPAN. I see a lot of improvement opportunities for the module. It could support incremental point generation. It could actually use a vector library like PDL instead of hand-rolling most of the vector manipulation code. It could use some tests beyond basic functionality tests.

But for the moment, I'm very happy with the fact that the code produces results I can use in creating random maps. And also I'm very happy with the fact that I have nice visualisation tools that enable me to visually verify the quality of datasets.

You can solve any problem by adding another layer of indirection

During the Easter weekend, I've started to write code supporting Firefox 4 in WWW::Mechanize::Firefox. So far, only some ugly parts of the restructuring of Firefox have broken WWW::Mechanize::Firefox. As my main application for it still runs Firefox 3.x, I have to support both versions, and especially not break any functionality for 3.x. So far, I could put most of the differences between Firefox 3.x and Firefox 4 into two submodules, WWW::Mechanize::Firefox::API35 respectively WWW::Mechanize::Firefox::API40. The constructor of WWW::Mechanize::Firefox does return either of these subclasses, depending on the version. This proves that you can't solve the problem of too many indirections by adding another layer.

As I have no real use case for Firefox 4 yet, I'm not sure whether there are more differences in when (and which) events fire during page load etc.. There is one nice glimmer of improvement for Firefox 4. I've found a Selenium test file that overrides many of the modal dialogs, so that I can now maybe implement ->proxy() for WWW::Mechanize::Firefox and suppress modal dialogs from blocking the whole application.

If you have Firefox 3.x and automate a critical application, don't blindly upgrade Firefox on your main machine. Firefox 4 and Firefox 3 will not peacefully coexist if you don't take preventive measures.

If you want to do a test drive of the upcoming version, take a look at the Firefox4 branch of the Github repository.

I love it when a plan comes together

Barbie blogged about the 11 million test reports mark, and gave a link to a dataset.

I had been sitting on my data "visualization" idea App::ffeedflotr for some time already, and had just started writing and publishing it on github. That dataset pushed me to write a short tutorial / showcase of the application.

The screenshots were produced by WWW::Mechanize::Firefox, the charts produced by the flot library, and the Javascript manipulation also came through MozRepl::RemoteObject. It's nice when your tools conspire to surprise you in a good way!

App-ffeedflotr - plot using Firefox and flot

Sometimes, I want a pretty plot of data. Excel is not always the tool of choice,
or the output of

while /bin/true done; ls $file ; sleep 1; done

Inspired by feedGnuplot , I wrote ffeedflotr.pl, which takes data and plots it in Firefox. So far, it is not really configurable. But as I have the API blueprint of `feedGnuplot`, I can easily/conveniently adapt it to…

Downwards compatibility, and a surprising find

I've just released a new version of WWW::Mechanize::Firefox, appropriating a new function of WWW::Mechanize, ->click_button. I used that occassion to also change the XPath queries sent by W:M:F to Firefox to be somewhat slimmer by changing queries for the element type from a global "OR" to a local test for the node name:

The old query to look for a <BUTTON> or <INPUT type="submit"> was

//button
|
//input[@type = "submit" ]

The new query is

//*[local-name(.) = "button" or (local-name(.) = "input" and @type="submit")]

The advantage of the new type is that the query engine does not need to search the DOM tree twice, and more importantly, that I can now easily ask for the "third button in this form", which I cannot do with two separate queries, as they don't necessarily return the buttons in the relative order in which they appear in the HTML+DOM.

All would be well if it weren't for differing Firefox versions. local-name(.) returns the element name. Except that in versions before 3.6, the element name for HTML pages is in ALL UPPERCASE, while since 3.6 onwards, it is in lower case. Of course, this is no real hindrance, as we can just lower-case() the element name. Except that Firefox does not implement lower-case(). It only implements translate(), which is uneasily similar to the tr/// operator in Perl. As you can imagine in your horror-filled dreams, the generated queries now look like this:

//*[translate(local-name(.), "ABCDEFGHIJKLMNOPQRSTUVWXYZ", "abcdefghijklmnopqrstuvwxyz")="button"
or (translate(local-name(.), "ABCDEFGHIJKLMNOPQRSTUVWXYZ", "abcdefghijklmnopqrstuvwxyz")="input" and @type="submit")]

... at least until I ditch support for older Firefox versions.

While I was copying the test suite of WWW::Mechanize, click_button.t, I saw that it launches a local HTTP server to test against that. Digging deeper, I found that it actually and actively uses a server that I once wrote to test WWW::Mechanize::Shell. This has really surprised me and made me happy. If people don't use my modules, at least my test infrastructure gets used by other modules in the same problem space, so I must do something right.

Patching XS modules

Today, I found out about a new release of Image::Thumbnail , which has support for Image::Epeg , a library coming from the Enlightenment desktop manager. Likely, epeg is quite fast, but it didn't build and test right under Windows. Conveniently though, Tokuhiro Matsuno maintains the module on github, so it was just a matter of forking and cloning his repository, and then trying to find out what made it break.

The breakage itself was three parts:

  1. A build failure where the symbols were not exported. It seems the epeg library wants -DBUILDING_DLL , which I supplied as a cc_optimize_flag through Module::Install, because I couldn't find a better way.
  2. Some test failures because binmode() was not used with the test image files. Easily fixed.

These two tests made it into v0.11 , released about 30 minutes after I told tokuhirom about the patches.

  1. The remaining problem was that a function call crashed Perl with

    Free to wrong pool ... at ...

This was due to a call to free(pOut) to release a memory block that the epeg library allocated for us. Unfortunately, the library uses malloc and free, and Perl redefines the two, so I had to find a way to make the XS module also call the original free() and not the redefined free(). Tye McQueen pointed out a working hack by making the compiler forget the redefinition:

#undef free

That undefinition would not have worked if there were other places below that line that still needed access to the redefined free(), but luckily that was not the case. The only snag was my limited knowledge of C preprocessor specifics, as I had some whitespace before the #undef and it was not picked up nor flagged as error. But after some trials, I also figured that out, and the pull request went out.

About Max Maischein

user-pic I'm the author of various CPAN modules. I'm also one of the admins of perlmonks.org.