Does Perl run out of memory?

Today I got an email from someone saying “I was told by a person who used Perl for computational genomics applications that it was running out of memory, so he switched to C++. What’s your thoughts on running out of memory in Perl?”

Just for posterity here is my reply (please note I’m no expert on this sort of thing and have never had the problem) was…

Now that I have "almost" fixed the remaining compiler bugs, I wanted to use the existing framework to enable the possible speedup by using types, esp. low-level internal types. C-style integers and double scalars are used internally in B::CC instead of full Perl IV's / NV's if declared as such, and thus greatly improves execution speed. Esp. for inlinable arithmetic and comparison blocks.

Only at the end of such a block (and only when really needed) the calculated C vars are written back to the perl pad. So I needed better Opcodes flags to define which ops read or write pads, and more possible optimization hints. Most of these flags should be added to core later, when the time will come to speed up not only the B::CC compiler, but also the internal perl compiler.

But how to declare types?

I released perl 5.13.10 today. I might write more about that later. But one significant change in it is that Perl now has many more regex flags.

So I wrote a short one-off script to find out what words I cound construct from the flags.

Now it just gives you one word that contains as many of the flags as possible, and gives you the remainder. What would be more interesting would be to detect cases where multiple valid words can be made from the flags. E.g. "mix" and "uploads". It just detected that by accident.

I leave that as an exercise for the reader.

Ignited by comments on IRC and in the Israeli Perl Mongers mailing list, I've sat (for 10 minutes), wrote and published a new Plack middleware: Plack::Middleware::NoMultipleSlashes. Let me explain what it is.

Apparently you cannot count (by RFC, even) that the paths http://mysite/ and http://mysite// will lead to the same place. This is very tricky, because it means that the framework you're using (Dancer, right? :) shouldn't clean those multiple slashes for you.

So, I figured "why not let Plack take care of it?". I wrote Plack::Middleware::NoMultipleSlashes (a single line of code, really), that cleans those multiple slashes for you, wherever they are in the path.

To use it with Dancer, just add the following to your config.yml file:

plack_middlewares:
- [ NoMultipleSlashes ]


Done!

Thank you, Plack! :)

BTW, if you still haven't gotten around to reading Franck Cuny's latest post, it's right here (PSGIChrome).

Win32::Wlan has now escaped onto CPAN. I've redone the whole structure. There now is Win32::Wlan::API, which is a very thin layer over the Microsoft Wireless API. Win32::Wlan itself provides an object that handles the initialization and deinitialization of the API and provides convenient access to the first (and highly likely, only) wireless connection of the computer.

Now I have to write a convenient "snapshot" program to invoke to "mark" a place, and another program to recognize those wireless snapshots and to associate them with a location. When a location is then recognized, this could trigger scripts that configure other programs for the proper proxy settings.

A lot of work is currently underway to improve CPAN. One longstanding issue is the handing of external dependencies (the "libfoo" problem.) As I understand it, although CPAN distributions may test for external dependencies, existing build tools are unable to install non-perl libraries, header files or compilers.

Resources to solve these problems are available, but in places not usually considered by CPAN toolmakers. I'm referring to native OS package repositories. While natively packaged CPAN distributions necessarily lag current CPAN offerings, they meet non-perl as well as perl dependencies, and are generally troublefree to install.

To give the example I know best, Debian's perl repository covers roughly 10% of CPAN, including much of CPAN's best. This enormous investment in labor, toolchain development and community building results in high-quality packages that even new users can install easily.

As a developer of a perl audio application available for Debian, I've encountered problems managing dependencies when administering systems that mix distro and CPAN sources, even when using local::lib.

The Problem

There is a lot of call for the ability to execute code at the end of runtime. An example of this is Moose when you call __PACKAGE__->make_immutable(). Wouldn’t it be nice if this were just automatic? Other places this can be useful are in testing frameworks such as Test::Class or Fennec. These testing frameworks use a ‘runner’ to run test structures defined in your tests.

The obvious (but bad) solution

Depending on your use-case for code that runs at the end of runtime, there are a couple possibilities. One possibility is to use an END block. Most people should know by now that END {} blocks are bad, simply read the perlmod section. There are just too many edge cases to account for, not to mention conflicting END blocks.

How Test::Class does it

Caching is a tricky business. Having just one kind of cache won't work, because the production environment will greatly determine the most efficient caching system. A distributed production environment would be best-served with a distributed cache. A smaller, single-server environment could use a simple shared memory cache.

Enter Jonathan Swartz's CHI module, the greatest Perl module to provide a unified caching interface. CHI is the DBI of caching: It presents an API, and delegates to CHI::Driver modules to perform the heavy lifting. It provides a layered caching system, allowing you to have a faster, more volatile cache in front of a slower, more persistent cache. It also provides a variable expiration time, preventing a "miss stampede" where all processes try to recompute an expired cache item at the same time.

By integrating CHI cache into WebGUI, we have the ability to provide any caching strategy that CHI can provide. We get Memcached, FastMmap, and DBI drivers (and more drivers can be written).

One of my long-term ideas is to make my laptop more location aware and to have something like cron except for locations instead of time. It will be useful to automatically reconfigure my network and proxy settings in Firefox and Thunderbird and to prevent information leakage. Also, my laptop can automatically synchronize its minicpan and git repositories with the mothership when it detects that a "fast" or "local" network connection is available.

As I'm not really interested (yet) in the physical location, it is enough for the laptop to know what wireless networks are visible to determine how it should behave. Later (or once GPS sensors become commonplace in laptops, or Perl becomes commonplace on GPS-enabled devices), adding GPS detection etc. will also become interesting.

I've pushed out an initial release (version 2.05) of HTML::Formatter which covers the incompatibility with current perls and RTF formatter initialisation.

I'll work through the other issues as I get the opportunity - but the incompatibility with current perl versions was the reason I took on maintenance, so I wanted to get a release with that fixed out quickly.

ActiveState have updated their PPM index page:

http://code.activestate.com/ppm/

As reported http://www.activestate.com/blog/2011/02/ppm-index-new-way-browse-perl-packages.

It is interesting to see which are the popular downloads and as an author the number of downloads of your own modules. They also have a nice chart of which OS's the module has been built for.

Not so sure that the example in the article - DBD::Mysql Failing on OSX - is such a good showcase! (looking at the report just seemed that mysql_config wasn't in the build servers path or something, so a setup issue, not a module issue).

But that aside it's interesting to see how far PPM seems to have come.

In a small script someone wrote at work I saw the indirect pattern of new Object, instead of the more correct form of Object->new. When I inquired (okay, I said "WTF?!"), he said that he just copied the synopsis of a module. Oh, right.. some of the synopses (plural of "synopsis", bet you didn't know that!) still have some outdated syntax examples.

I'm not gonna write about how we should all update our PODs to remove syntax that hasn't been (or shouldn't have been, at least) written for the last 10 years, even though I should! I wrote to talk about the reply my co-worker got when he opened a ticket asking for the synopsis to be updated, per my suggestion.

He got the reply "what is this, the syntax police?" Perhaps half-jokingly, but still problematic, IMHO.

Police? No. Neighbors, family members and friends? Yes!

I struggled with a problem where a given method would return an array of array refs of data, but the order (and sometimes presence) of array ref elements were sometimes slightly different. This is because this code needed to test real data and I could not mock the results. After giving this some thought, I realized I wanted something like the Levenshtein edit distance for data structures. Marcel Grünauer suggested that each element get assgined a unicode character. This solves my problem nicely with the following code ...

I saw a very ugly Perl code, and I think that is possible because no code review exist in project. But always we have a very little time for human code review. I search CPAN modules for basic level code review by machine. I hope that these modules help Perl people to avoid writing ugly code and help team leaders minimize time to code review. I try to collect founded modules in github repository and now want to describe what we know if all these test passed:

I might be working on a project for a friend shortly. They have lots of experience with developers who use Drupal - and I get the feeling (could be wrong) that you can semi-plug-and-play with Drupal?

So..
Want a wiki - use X plugin
Want a user system - use Y plugin
Want user profiles - use Z plugin
Want user gallery - use A plugin
Want a blog - use B plugin

So I started looking around, and there are plugins for specific frameworks, or standalone applications written in a framework, WebGUI CMS seems to have a lot of features but I want to use my framework of choice (and experience!).

Having been playing with Plack it got me thinking...

Just as Plack/PSGI sits between the webserver and your code, could someone (brighter than me!) come up with a standard for sitting between a framework and an app (blog/gallery/wiki/forum etc)?

Now that we've seen how easy Mojo::DOM makes parsing html, let's take a closer look at the css selector goodness it provides.

Here's a fairly verbose html sample for us to work with:

First, we initialize and parse the file:
use File::Slurp 'slurp';
use Mojo::DOM;
my $dom = Mojo::DOM->new->parse(scalar slurp 'some.html');

Getting all the articles' contents, of course, is easy:
$dom->find('li a');

But we can do better than that. Let's say we want only the article titles that have page anchors:
$dom->find('a[href^=#]');

Nah, let's get the article titles that link to external urls:
$dom->find('a[href^=http]');

How about only article titles that link to .net domains?
$dom->find('a[href*=.net/]');

We can also get the page anchors themselves:
$dom->find('div.article a[name]');

It could be that some articles have no text content; let's single those out:
$dom->find('div.article p:empty');

Hi Folks

I /still/ hate having to sign in twice in order to post. Grrrr.

Anyway, see this: http://vis.stanford.edu/protovis/

Examples: http://vis.stanford.edu/protovis/ex/

Slide mouse horizontally over this chart: http://vis.stanford.edu/protovis/ex/index-chart.html

However, let's not develop oodles on incomplete Perl interfaces, p-l-e-a-s-e. After all, we don't want another SVG debacle, do we?

I have my own blog, on which I post about Perl-related stuff along with other things. I also have a blog here on blogs.perl.org (obviously, as I'm posting on it now).

When I post something Perl-related, I'd like to post to both, as blogs.perl.org makes it more likely to be seen by the Perl community, many of whom probably don't follow my own blog, but I don't like the idea of manually posting it to both really. It's also a bit of a pain for any comments to be split between the two.

Solutions that come to mind:

• Automatically copying posts in the Perl category on my own blog to this blog somehow; or a script that posts a summary along with a link to the original post on my blog
• The above but in reverse; post here and have a summary & link posted to my blog
• Not use this blog, and attempt to get the feed for the Perl category on my own blog listed on planet.perl.org, and hope that helps make those posts visible to the Perl community

I can't be the only one who has their own blog but also wants their content on blogs.perl.org for better visibility within the Perl community - anyone have any suggestions?

David Golden recently posted regarding a comment from Leon Timmermans on IRC. Leon highlighted a problem when CPAN authors try to find information about CPAN Testers, and how they can request testers to do (or not do) something with a distribution they've just uploaded.

The page they are looking for is the CPAN Author FAQ on the CPAN Testers Wiki. Although there is plenty of information for authors, the page doesn't appear prominently on search engines when some searches for that kind of information.

As such, David has suggested that people tweet or post about the page, which includes this post ;) In addition, I'm going to look at adding this and potentially other useful pages as quick links on other CPAN Testers family sites. If there are specific pages you think should be mentioned, please let me know and we'll look at how best we can raise their profile too.

Cross-posted from the CPAN Testers Blog

Chiral Operators

In actual usage, the syntax of Perl's list operators is quite natural. Descriptions of that syntax, however, tend to be awkward.

The current practice is to describe this syntax in terms of "left precedence" and "right precedence". In other words, list operators are said to be chiral. I have problems with the Chiral Interpretation of list operators. The most serious of these: the Chiral Interpretation does not actually account for the behavior of expressions that contain list operators.

In this post, I assume you have a working knowledge of one or more list operators (examples are join and sort). The most authoritative account of the Chiral Interpretation is in the perlop man page.

Our Example

The rest of this post will use a single example: