Two frustrations I've had with a lack of good alternative to the msword integration code, and problems with interacting with the internals of a running firefox/xulrunner. I finally cracked open mozrepl, and the cpan module MozRepl and prototyped a bidirectional bridge between Zotero and perl.

The code is here, and should be a useful start for anyone who wants to do any bibliographic data mangling in perl. I'm not going to publish it to CPAN until I work out how to test it properly.

Anyway, the point of my post was that the organisers of German Perl Workshop were good enough to invite me to talk to them earlier this year, and at that time I obtained one of the wooden round tuits being distributed by The Perl Foundation. As a part of our trip to Borneo, we had to take some anti-malarial drugs. My wife and I had to take a whole tablet each. My daughter 3/4 of a tablet, and my son, 1/2. This meant some chopping up. Well TPF's mini-chopping board came in handy:

A nice portable mini-chopping borard, which with my habit of shaving with a disposable razor meant that I could prepare medication daily for the two weeks that we needed it with the minimal risk of lascerating myself. Big thanks to The Perl Foundation for helping me keep my programming fingers :)

There is a lot of data summarised in this report, and rather than producing a long turgid document with every possible analysis that I can think of, I thought that the better way to approach things would be to make a fairly short summary report so that people can ask questions, or request any additional information via the comments here, by email, or by grabbing me (kd) on irc.perl.org.

The most important outcome from this grant is that all of the R code I've written is solid enough ensure that we can run the survey again in a couple of years time, and quickly get longitudinal comparitive data out of it (so long as the questions remain reasonably similar). R isn't the easiest language in the world to work with, and I'm pretty pleased with some of the data management gymnastics I performed in getting the survey data management tools working reasonably well.

Oh yes, the data and analysis is freely redistributable under the same conditions as Perl itself :)

One of the things I need to do (for making reports) is to transform this:

 [ [qw/foo bar/],
   [qw/a b /],
   [qw/x y/], ];

into this tree structure:

 {
   'foo' => {
       'some_data' => 'lvl0',
       'children' => {'a' => {
           'some_data' => 'lvl1',
           'children' => { 'y' => 'leaf', 'x' => 'leaf' } },
                      'b' => {
                          'some_data' => 'lvl1',
                          'children' => {
                              'y' => 'leaf', 'x' => 'leaf' }}}}};

Being a nice golf problem I thought I'd ask on irc if there was a hacker better than me who felt like taking a look at this. ribasushi++ obviously had a little procrastination time available and wrote me a nice solution which I needed to make into a closure via a recursive subref:

There are two gotchas with this code. Firstly we need to weaken() the coderef before calling it in order to prevent a memory leak. The second problem is that we have to jump through some hoops to make the coderef in scope when $visit->(@args) is called inside itself. There are three ways of dealing with this. I could use Sub::Recursive, but the implementation is apparently non-ideal as it uses local, which is expensive. A more robust way via the CPAN is to use Sub::Current which provides a special function ROUTINE, which returns a code reference pointing at the currently executing subroutine. This is probably the correct way to do things. But the solution I plumped for (for now) is a little scope trick:

my ($visit, $v); 
$v = $visit = sub { ... $visit->(@args) ... };
weaken($visit);
my $data $visit->();

This is more fragile in terms of memory leak protection than using Sub::Current, but for my purposes is adequate for the present day. I have of course added a comment to the effect that I may need to upgrade to Sub::Current at some point in the future. (Thanks to mst++ for explaining this stuff to me in words that I could understand).

Now I have to work out how to get the real data into the leaf nodes :) ]]> tag:blogs.perl.org,2010:/users/holy_zarquons_singing_fish//23.1008 2010-09-14T00:05:47Z 2010-09-14T00:40:50Z

One of my many many annoyances is that track changes sucks, and it doesn't work well with the bibliographic software that I use. Unfortunately all my colleagues use word, so I don't really have a choice. But I've just started a new major piece of writing, and I want to be able to diff it properly.

So here's a little pre-commit hook that I made to commit plain text versions of my word docs in a separate tree off the root dir of the git repository. The only downside is that it's OS X specific because I use the built in textutil command to do the .doc to .txt conversion, but antiword or catdoc would probably be better.

Update: I decided I was not terribly happy with the output of textutil, so I changed it over to antiword, which works much better, both for layout, and with the field codes that my bibliographic manager uses. I also fixed a rather nasty bug that stopped it working with modified files.

Firstly the tl;dr: You can see the complete results for language usage at the survey website.

Here are the gory details (just a note: the <- operator is a bit like the = operator in other languages, but there's also a -> assignment operator too, which means you can do all sorts of clever things in 1 line of code):

What we asked was for the 5 programing languages that you use most, roughly in order of how much they're used. Then we asked where perl came in that list.

I'm doing this in R because it's optimised for statistical calculations. In this case though the operations required were a mixure of the kind of thing that R is good at, and the kind of thing that perl is good at. I decided that dropping out to perl wherever wasn't going to be a terribly good option, as I wanted to have the analysis file self contained. Anyway, here's the source code, let me talk you through it. The code is in R

So if we think about the original data file as if it's a database, then we have the language name stored in its rank column. So the most used language is in column language$most.used, the second most is in the column language$second.most and so on through to 5. Finally the position of perl is stored in a separate column so that eventually we'll do a shuffle along one procedure to splice perl into the list correctly.

Source code time:

load("10_language_info.RData") # grab data we sliced out of the raw data earlier
for (i in 1:5) data[,i] <- tolower(data[,i]) # make the text all lower case

Here we're normalising the list of languages. The raw data has about 350 languages. In fact this is actually around 67 unique languages, but people are inconsistent:

## languages <- sort(unique(unlist(c(sapply(data[1:5],levels)),use.names=FALSE))) 
# write.csv(languages,file="lang.csv")
#  some manual processing to normalise the languages list a bit 
languages <- read.csv(file="lang.csv")


for (i in 1:length(languages[,1])) data <- replace(data, data==toString(languages[i,1]), toString(languages[i,2]))
data <- replace(data, data=="rubh", "ruby") # a typo we missed in normalisation process

Now rather than having one column per rank, we need one column per language. The following code for each rank explodes the column into the number of columns that corresponds to the the number of languages mentioned in that column (a standard statistical procedure called creating a dummy variable). We do this for each rank:

library(dummies) # like use Whatever; in perl

# it's like a bomb in a mannequin factory!
l1 <- dummy.data.frame(as.data.frame(data$language_1))
l2 <- dummy.data.frame(as.data.frame(data$language_2))
l3 <- dummy.data.frame(as.data.frame(data$language_3))
l4 <- dummy.data.frame(as.data.frame(data$language_4))
l5 <- dummy.data.frame(as.data.frame(data$language_5))

# this is just to make the column names prettier for display
names(l1) <- sub("^data.language_.","",names(l1))
names(l2) <- sub("^data.language_.","",names(l2))
names(l3) <- sub("^data.language_.","",names(l3))
names(l4) <- sub("^data.language_.","",names(l4))
names(l5) <- sub("^data.language_.","",names(l5))

At this point each new data frame (the container we put the exploded variables in if you like) is just numbers 1 and 0 - we want to make this number the actual rank if it isn't zero:

l1 <- replace(l1,l1==1,1)
l2 <- replace(l2,l2==1,2)
l3 <- replace(l3,l3==1,3)
l4 <- replace(l4,l4==1,4)
l5 <- replace(l5,l5==1,5)

And we need a complete list of all languages mentioned.

languages.list <- unique(names(c(l1,l2,l3,l4,l5)))

This is the variable that's the number of responses:

cases <- rep(0,length(l1[,1]))

Then we need to make sure there are the same number of columns for each data frame:

for ( i in 1:length(languages.list) ) {
    if ( length (l1[[languages.list[i]]]) == 0 ) {
        l1[[languages.list[i]]] <- cases
    }
    if ( length (l2[[languages.list[i]]]) == 0 ) {
        l2[[languages.list[i]]] <- cases
    }
    if ( length (l3[[languages.list[i]]]) == 0 ) {
        l3[[languages.list[i]]] <- cases
    }
    if ( length (l4[[languages.list[i]]]) == 0 ) {
        l4[[languages.list[i]]] <- cases
    }
    if ( length (l5[[languages.list[i]]]) == 0 ) {
        l5[[languages.list[i]]] <- cases
    }
 }

And then we add the 5 data frames together in a matrix addition operation (a data frame is basically a special matrix). Note that we need to make sure that each data frame returns the columns in the same order.

all.langs <- l1[names(l1)] +l2[names(l1)] +l3[names(l1)] +l4[names(l1)] + l5[names(l1)]

perl <- data$where_perl_belongs_in_list

insert.perl.order <- function(row.idx) {
    x <- all.langs[row.idx,]
    y <- perl[row.idx]
    change.logical <- which(x >= y);
    all.langs[row.idx,change.logical] <- x[change.logical]+1
}

all.langs$perl <- perl #append it to the data frame

We're at an important point here, because all.langs can be glued to other R data structures created in other parts of the survey, so at some point we can work out which programmers are the smartest perl programmers (or something like that).

Next up there's some jiggery-pokery to make sure that R knows that there are six levels of variable in the data frame, otherwise it will only report on the actual counts, and won't report zeros:

for (i in 1:length(names(all.langs)) ) all.langs[,i] <- factor(all.langs[,i],levels=c(1:6) )

This single line of code generates a counts for all languages:

lang.summary <- sapply(all.langs, summary)

In case you hadn't realised R is a functional language, and a variant of lisp.

Finally we want a report, so we make a new data frame that contains the summary statistics:

report.df <- data.frame('Most used'=integer(),
                    'Second most'=integer(),
                    'Third most'=integer(),
                    'Fourth most'=integer(),
                    'Fifth most'=integer(),
                    'Sixth most'=integer(),
                    'Non-users'=integer(),
                    'Total-users'=integer(),
                    'Percent users'=numeric(),
                    'Mean Rank'=numeric()
                    )

And we have to iterate through the lang.summary matrix to append lines to the report data frame:

for ( i in 1:report.rows ) {
    this.lang <- names(lang.summary[1,])[i]
    this.counts <- lang.summary[,i]

    # UGLY and possibly FRAGILE hack to remove non-perl users from counts
    this.counts[7] <- this.counts[7] - lang.summary[7,dim(lang.summary)[2]]

    this.counts[8] <- sum(this.counts[1:6])
    this.counts[9] <- round(this.counts[8]/sum(this.counts[1:7]) * 100,2)
    this.counts[10] <- round(sum(this.counts[1:6]*c(1:6))/sum(this.counts[1:6]),2)
    report.df[i,] <- this.counts
    rownames(report.df[1,]) <- this.lang
}

rownames(report.df) <- names(lang.summary[1,])

Note the comment in the above code. Getting all of this working just right was very fiddly and has taken more than three times my allocated time budget on this for the last three days. You'll also notice that the code is very procedural, for a functional language. This is basically because statistical computing is often long periods of exploration on the command line (or GUI), followed by consolidating what you've done into a script. The script is basically a duplicate of the procedure you went through on the command line. On the other hand, it's done now, it's reasonably robust, and it's replicatable for future runs of the survey.

Now that I've worked out how to write the report in a reasonably replicable way (i.e. minimum code differences between revisions of the survey - so running and analysing the 2012 survey should be much much quicker and easier), I should be able to get through a section every day between now and when it's done, then I can call the grant finished!

For the data management end of things I use a super-simple SGML format to tag chunks of text that I'm interested in, and use my Text::TranscriptMiner library to retrieve data. I use Git for version control, as a kind of high granularity lab book. I also have a web application which deals with the data visualisation side of things.

One of the things about qualitative analysis software is that there's a lot of post-processing handling of the data, and a need to write summaries (technically called memos) of the raw data. To that end I need to think about improving the visualisation tools in the web application, and that suggests to me editing data directly in the browser. This is super easy with the following javascript function:

document.getElementById("my_id").contentEditable = "true"; .

So here's how to make a crudely editable section of a web page (the content div inside the form), and return it to the back-end web application:


<script type="text/javascript">
$(document).ready(function() {
document.getElementById("content").contentEditable = "true";
$('#data_container').submit(function() {
$('[name=data]').val( $('#content').html() );
});
});
</script>
<form id="data_container" action="doit" method="post">
<div id="content">
<h1>hi there</h1>
</div>
<input type="hidden" name="data" value="default" />
<input type="submit" value="submit" />
</form>


At which point, we have well structured html returned to the web app, with which I can use a tool like Web::Scraper to extract the data and put it somewhere sensible for the back end to retreive at a later date.

Sometimes you're lucky and a proxy url of the form

http://user:pass@my.proxy.server:8080

Anyway, here's the way to use LWP::UserAgent behind an authenticating proxy:


use LWP::UserAgent;
my $ua = LWP::UserAgent->new;
my ($user, $pass) = qw/user pass/;
$ua->proxy(['http', 'ftp', 'https'], 'http://my.proxy.server:8080/');
my $req = HTTP::Request->new('GET',"http://www.google.com");
$req->proxy_authorization_basic($user, $pass);
my $res = $ua->request($req);
print $res->content;


And here's how to do it with WWW::Mechanize:


use WWW::Mechanize;
my $mech = WWW::Mechanize->new();
my ($user, $pass, $proxy) = qw(user pass http://my.proxy.server:8080 );
$mech->credentials( $user, $pass);
$mech->proxy('http',$proxy);
my $url = 'http://whatsmyip.net/';
my $response = $mech->get($url);
print $mech->content;


I'm not really sure how you'd go about accessing a basic auth protected page with the WWW::Mechanize method mind you. It'd also be nice if (in both cases) whether to use a proxy, and the username and password could be handled during the call to

->new

There appears to be some very good news in that people use and are enthusiastic about new versions of perl, and the cutting edge perl modules like Moose, Catalyst, DBIx::Class and so on.

Most of the analysis and graphs were done with R, a little Perl, and unfortunately for the CPAN and programming languages questions I had to result to a mix of Perl and manual munging with a spreadsheet.

The survey has generated a very large amount of information, and now that I've got an initial cut of the data analysis, I'll be writing a formal report, and investigating any interesting patterns I find on the way.

All the data is available github. The presentation slides are available from here (sorry about the small font size in some of the legends).

Most importantly I'll be able to release the (lightly processed) raw data for you by that time as well.