Adapting PDL to a Big Data Landscape

Note: although this article is directed at current PDL users, I would particularly appreciate the opinion of Perl users who are considering using PDL. Does my assessment seem accurate to you?

I was just watching a few of the talks on youtube from from YAPC::NA that I wanted to attend in Madison but could not because I was busy (writing my talks) or attending other talks. And it reminded me of the revelation that I had at YAPC. Although I am not looking for a job, I spoke with the sponsors at their job booths, just to get a feel for what's out there. Is it possible for a Perl programmer to get a job doing real data crunching? The answer, happily, was "yes".

A distributed design challenge

I have a distributed design problem that I thought Monks might find interesting and perhaps be willing to offer suggestions on.

The Setup

I'm writing real time bidding software. It works like this:

  1. My software receives a request to bid on an auction
  2. I return a bid on said request (I must respond in 85 to 100 milliseconds)
  3. Later, I receive a notification on whether or not I won the auction

Point number 3 above means I don't find out if I've won a request until shortly after I've made the request.

Currently, on a server farm of about 36 boxes, we're receiving roughly 400 bid requests a second, or about 11 requests/second per box. This is an extremely light load and will likely increase by one or two orders of magnitude when the system goes live.

We have various campaigns, each with individual daily and total budgets that cannot be overspent (a bit of wiggle room is allowed). The servers run separate databases and the only shared data source we have is a Redis server with two slaves, though we could implement others.

So that's the setup. Now on to the problem.

Net-A-Porter.com sponsors YAPC::Europe 2012

We welcome Net-A-Porter.com as a Gold Sponsor of this years' YAPC::Europe. Without the help of all our sponsors, the event would be almost impossible... Thanks!

net_a_porter.jpg

Brainturk - Cognitive Games

Better, Faster, Funner - Part 1

I've recently started working on a new web app project using all sorts of latest-and-greatest techniques and libraries, needless to say I choose Perl to build it out. The architecture of this application is different from what I'm used to but I like the direction I'm headed, everything seems to be in order, so I'd like to share my thoughts, struggles and accomplishments.

The WAF (web app framework) I choose to develop around this project is Dancer, although I've been away from Dancer and its community since before version 1 was officially released I decided to use it because, quite-frankly, it has nice and robust documentation, sensible defaults, and does exactly what I need it to with minimal code. You may have noticed that I neglected to mention the framework's plugin ecosystem, which IMHO is not preferred for large/growing projects.

The Hub

Better, Faster, Funner. Part 1

Better, Faster, Funner. -- Part 1

I've recently started working on a new web app project using all sorts of
latest-and-greatest techniques and libraries, needless to say I choose Perl to
build it out. The architecture of this application is different from what I'm
used to but I like the direction I'm headed, everything seems to be in order,
so I'd like to share my thoughts, struggles and accomplishments.

The WAF (web app framework) I choose to develop around this project is Dancer,
although I've been away from Dancer and its community since before version 1 was
officially released I decided to use it because, quite-frankly, it has nice and
robust documentation, sensible defaults, and does exactly what I need it to with
minimal code. You may have noticed that I neglected to mention the framework's
plugin ecosystem, which IMHO is not preferred for large/growing projects.

The Hub.

Want The Game Crafter to get sposoring? Use your Facebook to vote ASAP!

The Game Crafter is a Perl-shop that allows anyone to create their own board game or card game. The website and the backend code is in Perl. It's providing Perl jobs, and more important, Perl initiative. It shows not only that Perl is alive and well, but that it creates some of the more fun things that are going on.

They were at YAPC::NA 2012 and provided game night with lots of board decks (many of which were given away) and board games for all of us to play.

There is a competition to get $250,000 (!!) of sponsorship for startups. If you have Facebook, you can vote for The Game Crafter to get this money.

Here are the five step instructions.

They have only two more days of voting. Get to it!

I Want My Objective-C

The other day, as they idea of the defined-dereference operator (~> is the favorite at the moment), I proposed the idea of implementing a pragma that changed what Perl does when it's faced with a dereference of, or a message sent to, undef. Right now, of course, it dies.

This bit me hard this evening as we put up some new code; I have a Mason template that calls a series of methods to get a particular piece of status information to be shown to the user. I had bullet proofed it, or so I thought.
my $foothing = $current_object->foo;
if (defined $foothing) {
    my @barthings = $foothing->bars();
    if (defined $barthing[-1]) {
        my $answer = $barthing->baz->status;
    }
}
Yep, $barthing->baz came back undef, and an error page instead of my actual page.

About blogs.perl.org

blogs.perl.org is a common blogging platform for the Perl community. Written in Perl and offering the modern features you’ve come to expect in blog platforms, the site is hosted by Dave Cross and Aaron Crane, with a design donated by Six Apart, Ltd.