Over the past few of months, various fixes and improvements have been made to the Builder process, which builds the pages and support files on the CPAN Testers Reports website. As mentioned in the last summary, this has made a noticeable improvement in the performance of the server. However, there were further fixes and enhancements planned.
Several updates ensued during January, not just for the Builder, but across several parts of the eco-system, both to speed up processing and to reduce the amount of file and database access. Previously several processes recreated their current view by reading the full result set from the database. As we now have over 10 million reports, this can take more time than is reasonable. With the use of the JSON files to record a snapshot, we can now start from a known point, thus meaning we only need to scan a few thousand records. This method has been utlised in other processes, even with smaller database tables and datasets, and the performance improvements have been significant.
I just got back from FOSDEM 2011 last night and had an absolute blast, especially on Sunday when I spent the whole day in the Perl devroom. Thanks to everyone who came!
I gave a talk called Introduction to writing readable and maintainable Perl and I thought I'd share my slides here.
I missed the Parse::Stallion and most of the Intro to Dancer talks because I spent it talking with olegm and perigrin.
I really would have liked to catch them but it is always interesting to talk with folks.
perigrin is a fellow player of Lacuna Expanse and it was good to meet him.
I then mostly listened to olegm and perigrin compare notes about putting on a workshop, relative costs...generally what perigrin was going to talk about after the Intro to Dancer talk. :-)
The most interesting comment was when perigrin said he would like to see a conference/workshop in each state in the US. It seems like every country in Europe has one so why not each state.
A very compelling idea, especially when you consider that Perl Oasis costs about $2,000 to put on an attracts around 40 attendees.
They told me it was evil. They told me it was not maintainable. They were right.
But the alternatives were painful. They were clunky. They required me to change the way I approached HTML. They required me to abandon the hipness of css selectors I had embraced with javascript libraries, and ignore the many years spent perfecting my css-foo.
HTML::Parser, HTML::TreeBuilder, I'm sure you're brilliant in your own way. I'm sure you have conquered many lands, and for those who wanted to adapt to your mindset, you brought much happiness.
I wanted a simpler way. jQuery taught us that css selectors are that better way.
Fortunately, Mojo::DOM sprouted up out of the land of cookies and rainbows and unicorns.
Seriously, have you ever seen HTML retrieved, parsed, and processed so nicely in Perl?
Dancer's engines is really cool. You wanna know how cool? Here's an example.
If you like Mojolicious' templating system and you want to use it with Dancer, our interchangeable templating engines allow you to use a template engine of Mojolicious, if it exists.
And if it doesn't exist, you can write it. Oh wait, someone already did!
Check out the source to find out how silly easy this is.
So, if there's another template engine you want with Dancer, try to write it! If you have issues, talk to us on IRC (#dancer on irc.perl.org) and check out other template engines.
For over a month I have been working on a little project called QueryHunt.
it is a questions and answers search engine powered by yahoo BOSS.
Why did I decide to use Perl for this project?
When I started working on this project I had just started to learn Perl, I was planning to write the site in PHP.
Perl had a lot of promising features that would make the development process easier in the long run
some of this features are
Easy XML parsing
Regex support (Vital to discard results that were neither a question or an answer.)
DBI (in case I decided to give PostgreSQL a try compatibility issues wouldn't be an obstacle)
so for obvious reasons I ended up using Perl, The site has now been running for about 2 months and I'm more than happy with it.
I'm not on vacations anymore so I don't get to work on it so much but I have a lot of ideas and I have started to work on a home-made crawler that I will be more than happy to share with Perl programmers.
I will keep updating this post to write about obstacles I face and other things that might help programmers that are working on similar projects.
Following the lead of Mike Doherty (DOHERTY) I've moved my own Dist::Zilla Author PluginBundle into a new namespace:
Dist::Zilla::PluginBundle::Author::OLIVER
Having Author PluginBundles is a great system for saving me time and allowing others to build my modules with ease. However I had to agree to Mike's point that polluting the Dist::Zilla::PluginBundle:: namespace wasn't so cool. It also makes things more clear in the dist.ini file:
[@Author::OLIVER]
If you have an Author PluginBundle, please consider moving it into this new namespace. You can of course keep the old one around, in parallel, until code using it has moved off to the backpan.
Dave Rolsky has written a compelling post on how not to write a changes log. It's ironic (or is it? the meaning of "irony" is illusive) that while I have much criticism for the changes log of others (and have commented on them to people in the past), Dancer's changes log is not up to par with what I think it should be, nor what Dave thinks it should be (which is close to what I think).
Understanding Dancer's changes log
Dancer's changes log has two primary goals: mark down changes for Dancer users, and giving credit to the people who do it.
This means that in each version we have who did what change, and additional credit for whoever helped in any way.
While the current format of the changes log is not optimal, we cared more about maintaining the current state, and did not optimize it. Truth be told, it was a bad habit and none of us advocate the current style be remained.
Yes, it's true. Even though I live on the southern edge of the Australian mainland, the floods have hit here. I'm in Melbourne, the capital city of the state of Victoria.
Last night, about 9 pm, the floods from other parts of my suburb, Murrumbeena, backed up, and drowned my car to about halfway up the windscreen. The car was parked in the street, in front of the house.
Luckily, the house is on a slight slope down to the street, so the floodwaters stopped about the height of 1 (one!) brick below the floorboards.
This morning I was on the phone for about 10 minutes to roadside assistance (a commercial service) who say that since water got inside the car, it's an insurance matter. So, they transferred me to another department, and after another 25 minute wait I lodged a claim.
Since the car won't start, it's going to be towed to a local crash repair shop. I've been warned by the insurance company that there's no guarantee it'll be salvagable.
He likes Perl a lot, in the last few years there has been a fundamental change in Perl. Things like the modern Perl movement, chromatic's Modern Perl book, Moose, Catalyst and DBIx::Class changed how we can use Perl.
Even with all that Perl has an image problem.
What are the Perl companies we trot out: LiveJournal, Ticketmaster, Amazon and SixApart.
He noted that some of them are moving on to other languages for their new development and that some have written Perl in such a way that makes one not want to ever use Perl again.
Nick then presented some of the research he was done. Looking at sites like LangPop, doing searches in sites like Craigslist and sending out a survey to the many people and groups he knows.
He had lots of interesting quotes and such that I didn't put in my notes. I'm hoping he'll put the slides up on the Frozen Perl site. Looks of data to look at and ponder.
If you installed WebGUI 0.9.0 back in August of 2001 (the first public
release), you've had a stable upgrade path through WebGUI 7.10.8 (January
2011) and beyond. Plainblack.com has been through every upgrade for the last
10 years, a shining bastion to our upgradability.
A WebGUI 7.10 user would not even recognize a WebGUI 6.0 database, much less
the database used by the 1.x series, but slowly, gradually, our upgrade system
brought new features to every WebGUI site that wanted them.
Our upgrade.pl script would check for docs/upgrade_*, compare version numbers,
and then execute the .sql and .pl scripts in order until there were no more
upgrades left.
I am honored to be sponsored by PEG, and I would like to thank them for it.
A team of Dancer core developers (Alexis Sukrieh, Franck Cuny, Damien Krotkine and myself) will be having a mini-hackathon this FOSDEM. This is made possible since we will all be staying together in the same apartment for the duration of the event.
We will focus our efforts on merging Github Pull Requests and closing as many tickets as possible. New features might be worked on, but it is not part of the official plan. We leave room for improvisation. :)
I want to thank everyone who pushed commits and changes into Dancer. I've been very surprised (yet thrilled) at some of the new faces we've been seeing on our IRC channel (#dancer on irc.perl.net) and in our pull requests. While some of these were merged on the spot, others were waiting longer in the queue. This is what we will try to focus on.
The next post will include another issue which will be worked on during the hackathon, which is the Dancer changelog.
If you're arriving at FOSDEM, we have a Perl room and a Perl booth. Feel free to stop by, say hi, catch a few interesting talks and jibber-jabber with us!
The CPAN META specification includes support for "resource" links to homepage, bug tracker, mail list, source code repository, and so on. These will appear on search.cpan.org if you ship a META.json with your distribution.
Dist::Zilla's MetaResources plugin allows you to set these links in the distribution config, but I wanted something a little more automagical. This can be done because I've set up my GitHub repositories with consistent naming and layout.
So my new plugin, Dist::Zilla::Plugin::MetaResourcesFromGit, is a drop-in replacement for the standard MetaResources plugin. It automatically generates four resource links based on the name of the distribution and the remote git repository settings. Simply use the plugin in your dist.ini file:
I need to add the _raw and _str variants for everything, so not having them will always be wrong, or at least a mystery
<confound>
This is a great mystery. But I am speaking of MIME and delimiters.
<rjbs>
Email: the E is for Ephesians?
<confound>
Therefore, since we are surrounded by such a great cloud of RFCs, let us throw off every feature that hinders and the proprietary extensions that so easily entangle.
<rjbs>
confound: Where were you when I divided the tokens from the whitespace?
<aristotle>
confound++ # ephesians
<rjbs>
thus marking the first time anyone on IRC was ++'d for Ephesians
<rjbs>
Encode always; lookahead without ceasing; in everything, be strict, for this is Postel's will for you.
<confound>
Then he will say to those on his left, "Depart from me, you who are marked as spam, into the limitless bitbucket prepared for penis enhancement and chain forwards. For I was using mutt and you sent me a Word document, I sent you mail and you greylisted me, I wrote to the list and you vacation autoreplied, I made an announcement and you clicked 'reply all'."
err...I have to say, (I can't remember the exact time I met this version of quicksort), ever since the Haskell version was born, there were (and still are) various implementations available in various languages. This version of quicksort is a very nature "translation" of what the algorithm is:
function quicksort(list) {
select a pivot;
return (quicksort([x|x<pivot]), pivot, quicksort([x|x>pivot]));
}
Today, I met this version again in here(reddit.com). For convenience, let me repost the Lisp code:
Steven Lembark gave a talk about wrapping installed subroutines.
It was noted on irc that this was the cotinuation/sequal of another talk. The jumping off point is having to work in some code and it has the following at the top of every program/module
use warnings;
$SIG{ WARN } = sub { die };
Now think about what that does...
Yes, you are now screwed.
Especially if you want to do any benchmarking or deal with some other modules.
You can muck about with the symbol table and such but one solution is the Symbol module.
Lembark chose this because it is part of the core/standard libraries (a predication of his talk). It cleans up some of the mess of dealing with globs/symbol table weirdness.
It was noted on irc that Package::Stash is a bit cleaner to use (at least its API is better).
Regardless it was yet more of Lembark showing some of the weird things that you can do (and some times have to do) and why it is better to find a module to give you some distance from that mental mayhem.
As always, he shows off interesting bits of the Perl solution space.
Friday and Saturday have been very productive days for Dancer. We wanted to write up this blog post yesterday night but we were waaaaay too tired for that. Instead, you get it this morning while we're sitting at a great Perl 6 talk by Gabor Szabo.
Friday we met up. I tried to wait for Franck at the train station and got lost... several times. He found me in the end and we went to our hotel room with his co-workers. Sukria and Dams arrived later. It was very exciting to meet the guys I've been working closely with for a while and haven't even met in person yet. Free software sure is nuts! :)
We couldn't fix the internet at the hotel (one cable, no wireless) so we spent a lot of time on discussions about important things. While we do not like bureaucracy, some things had to be sorted out and talked about. Here is a short list of things we've settled:
