When you create a new post, you start off in the abstract, cunningly called the body, and it's empty. Instead it could have some boilerplate in it, such as:

This is the abstract for your post. Put one or two short paragraphs here, summarising your post, and then put the rest of your bullsh article in the extended section.

This would serve as both reminder and education.

Idea #2: start off in extended

Instead of starting off in the abstract, start the blogger off in the extended section, and put a one-line abstract in, along the lines of:

Another gripping installment from insert name here.

This could be made entertaining by randomly selecting the boilerplate from a file, à la fortune(6).

Idea #3: Change the names of body and extended

Change the labels body and extended to abstract and body.

Idea #4: Limit the length of the abstract

Limit the number of characters that can go in the abstract. When you get close you'd be warned. It could either be a hard limit (force everyone to split into short abstract and longer body), or just a soft limit, aimed more at educating users, and encouraging them to provide a separate abstract.

Suggested by sillymoose (see comments).

While I'm here ...

Thanks to Dave Cross and Aaron Crane, who run this service for us. I'll be submitting these ideas to them, but am posting them here in case / hope they prompt additional (better?!) ideas.

This would be a "front page for the perl community". It has feeds from a range of Perl sites, and data that would be provided by users: details of your perl blog, talks you've given, perl sites not currently listed.

This is kind of a mashup of perl.org, metacpan and perlmonks. As MetaCPAN is to CPAN, I think this could be to Perl: MetaPerl. It could even be the front page to MetaCPAN, and MetaCPAN already provides some of the information needed, but I don't think it fits within MetaCPAN's mission. Olaf? To be honest, I think perl.org is the most appropriate place for this.

It's an idea and an experimental prototype, so it's not on CPAN, it's on github :-)

The perl sites, blogs, events, and talks are all just JSON lists, if you want to add things that are missing.

At the moment I'm going with theperlhub.com for the name, or perhaps perlhub.org.

Here's a first cut, from trawling various blog posts, comments, and personal mulling. A blog isn't the best format for this sort of exercise, so I've created a google doc, which you can all edit. I'll sync any updates in both directions.

I'm deliberately not describing possible solutions here, just shooting for a succinct list. There will be overlap, and some that are really just symptoms of others. You may not agree with some, but at this point I'm hoping more for the union than intersection. I'd be happy to hear better names for some of these too!

• Discoverability: It's hard to find all the modules that are relevant for a particular need.
• Whipupitude: People don't look for existing competing modules before releasing a new module, as it's often quicker to just write the module you need than find one (or people do look, but don't find some of them, due to the discoverability problem).
• Best fit: Hard to tell which module is the "right one" for a need.
  Yanick on alternative dist recommendations; TIMB on suggested alternatives.
• Me too: When people perceive a problem with an existing module on CPAN, they're often quicker to release their own module than to work on "fixing" the existing one. This may be down to the orphan, communication, or resistance to change problems.
• Orphans: Too many modules are essentially unmaintained, but not explicitly so.
  brian d foy on ADOPTME user.
• Communication: Contacting authors of modules is sometimes near impossible, if they've changed email addresses but not updated PAUSE. This is different from the Orphans problem: I've had situations where when I've eventually tracked down the author, they're happy to give me co-maint.
• Barrier to contribution: It can feel / be hard to contribute to a module, and feels even more so if you just want to submit a small change (eg doc). [annocpan]
• Maturity: Hard to identify the maturity of a module (and maturity is in the eye of the beholder). There can be massive differences in performance, but you've no way of telling that unless you benchmark all the alternatives, with representative data.
• Resistance to change: Hard, or at least feels like it is, to rename or delete a module. Partly because of the unknown usage problem.
• Unknown usage: no way to really know whether your CPAN module is being used, unless it's by other CPAN modules, or if someone emails you.
• Land-grab: Namespaces are first-come, and once you've grabbed a namespace, it's essentially forever.
• Convergence: multiple solutions to a problem can inspire new approaches, but there's not much force for convergence, and attempts at convergence can often just add one more module to the mix.
• Duplicated dependencies: If you use a number of dists from CPAN in the same app, you can easily pull in multiple implementations of the same thing. Eg base vs parent vs superclass; exception handling; constants; OO framework; etc. you might argue this is a perl problem not a CPAN problem :-)
• Private: Many modules are used mainly / only for personal / semi-private needs.

Added

• Poor design: Many modules are poorly designed or over-engineered, solving problems in a specific way for the author that hinder more general reuse (thus prompting fragmentation)
• Dependency indifference: not caring about what your dependencies are, and as a result how many modules might be ultimately pulled in by someone using your module.
• Community tools: lack of community tools and therefore engagement, reviewing etc.
• Lack of cooperation: besides a few large / well-known dists, most modules are developed by single author with possibly a small number of contribut{ers,ions}. (as of November 2012, 79% of modules had a single PAUSE id against them; another 10% have an owner and 1 co-maint.
• Contribution undervalued: uploading your own module often given a disproportionately higher value than contributing to someone else's module (fixing a bug, submitting an addition, writing tests, documentation, etc).

What else?

If it's not clear, I'm not trying to say that CPAN's shit: most of these problems are born out of CPAN's success.

This addresses just one of CPAN's problems, which have also been raised today by Brendan Byrd.

There are many modules on CPAN which appear to be the result of some experimentation. Once the author has demonstrated their point, (s)he loses interest, and the module lurks on CPAN, waiting to catch out the unwary.

I've reviewed a number of module categories now, and in a number of them I've hit such experiments. I've had email with some of the authors, who've admitted the heritage, and often comment "oh, I forgot about that module".

The problem with these modules is that they just reduce the signal-to-noise ratio of CPAN, and make life harder for users, particularly those new to Perl and CPAN. Consider the following:

• If you want to define some constants, there are 21 modules.
• If you want to perform some kind of run-time loading, for example of plugins, there are at least 43 modules you might consider, for about a dozen use cases.
• If you wanted a module to find dependencies of your code there are 26 modules, but only about 5 distinct use cases.

In all of these categories there are modules that were experiments that never went anywhere.

Let me be clear: I don't want to suppress experimentation - it's a key contributor to progress. I imagine Moose started off as a wee experiment.

Proposal

If you've been experimenting and would like to share, either (a) to show "hey, look what I did", or (b) you're just not sure whether anyone else would use it:

• Put the module on github. Sure, there are other places you could put it, but github is fairly well linked into the Perl ecosphere, and cpanminus 1.6+ can install modules direct from github:

          cpanm git://github.com/miyagawa/Plack.git

• Write a blog post about it. If you don't have a blog, or your blog doesn't have a wide readership, then consider posting to blogs.perl.org as well. If you link to your blog, you might even gain some more readers.
• Discuss your module on PrePAN.
• If there are existing modules on CPAN which are similar, you could email their authors, asking whether they might link to your github repository in the SEE ALSO section of their doc. Ok, I admit your success rate might not be good here.
• You could annotate these modules on annocpan.

But if many modules start as experiments, how do you decide whether / when to release to CPAN? Just apply some common sense. For example, if you or anyone else starts relying on the module, then it's time for CPAN.

How might this idea evolve?

• If the name weren't already taken, PrePAN could be a place for uploading experimental modules. Maybe PrePAN could evolve to become that as well?
• If MetaCPAN indexed all perl dists on github, it might not include them in search results by default, but could say "N additional things on github matched your query, click here to include them".

You could also consider deleting any experiments you already have on CPAN: cpanminus can install from backpan. Check with the reverse dependencies service first. Deleting modules from CPAN is worthy of a separate post, but Brendan Byrd might beat me to it.

Revisited, 2 days later

I think have should have more clearly defined what I mean(t) by experiment!

There are a number of situations when I'd consider a module to be an experiment, but the classic example (for me) is where you're writing a module with no intention to use it in any real code. This might be to see whether something is possible, possibly trying to (ab)use Perl in some unexpected way. Such an experiment may obviously lead to something unexpected and useful.

Another category, but less clear-cut for me, is when you're creating something, but you're not sure exactly what it is you're creating, and whether there might be a module on CPAN already. Often the namespace(s) will change, and you may drop it anyway. Typically at this stage I just don't share it, because I don't want to worry about namespaces (even though you can free them after you rename, I know), but I've had a couple of cases where people wanted to play with the code anyway. See below.

Some examples might clarify, and might help me refine what the hell I'm talking / thinking about!

Not experiments

• Someone new to Perl writes a module which they're using at work. They proudly upload it to CPAN. They might have no idea whether it would be of interest to anyone else.
  As an aside, one problem (with the current toolchain) is that authors aren't encouraged / helped to find modules on CPAN that might serve their needs, or which they might be able to take over and evolve (commented on by brian d foy in a comment on Brendan Byrd's post on problems with CPAN). A topic for another day.
  
• Karen's post describing a dev release of Test::Warnings. Karen describes the implementation as experimental, but I don't see the module as an experiment — it's clearly written to meet a need, and will be used.
• Someone writes a module that's a complete hack (deadlines, we've all been there), but which they imagine they'll probably get around to doing a 'proper' version of eventually.
  Aside: it's not very easy to tag your dist with maturity, as others have noted. You can't just look at the version: you'd miss Net::HTTP::Tiny 0.001. 10.7% of dists on CPAN have version 0.01 or 0.001. And you can't just look at reverse dependencies: Net::HTTP::Tiny has none — I tend to use it in some of my scripts, and HTTP::Tiny in modules.
  
• Someone has a (possibly slightly crazy) idea for a module, which addresses something they see / have as a very real need, and which they think might pan out. I think this comes down to personal definition of experimental, and how you like to play that out, but if there are others already interested in joining in, then I'd always err on the side of CPAN.

Experiments

Where you draw the line between experimental and not is a personal call. From now on before I upload a new module (and as you can see, it's not something I've done many times), I'll just ask myself whether this is experimental, by my personal definition. If so, I'll release it to GitHub, and possibly describe it in a blog post. And if someone else starts using it, I'll put it on CPAN.

I consider the following to be experiments:

• I was looking for a tool to graph dependencies, and started searching for modules. I didn't find anything that met my needs, so knocked up a module. I'd already found a handful of modules, so started writing a review, and decided I wouldn't put my module on CPAN until I finished the review, in case I found a module I was essentially duplicating. As I progressed I kept finding more modules, and others pointed out modules in namespaces I'd not even considered. Still I found nothing in direct competition with my module. But the 23rd module was, and so I may not ever release my module, but either submit changes, or refactor it as a helper module. If I find myself in this situation again, I'll put it on GitHub.
• I have various modules and scripts I use when writing reviews. At some point I plan on releasing them to CPAN, but until recently, I seemed to tear them apart every time I wrote a review, including a change of namespace. I'm hoping to stabilise them soon, and will then put them on GitHub, and then ask Leo (and anyone else interested) to comment on them. I probably won't put them on CPAN unless / until someone else writes a review using them, which may be never, and that's fine.
• I suspect that Module::Hash is an experiment. Someone new to Perl might come across this module and think "This Toby guy seems to know what he's doing, the module is recent, it's well documented, so maybe this is the modern/new way to load modules at runtime". To me this seems like a good candidate for "GitHub and blog post", and I can see the blog post generating comments, which may in turn lead to a CPAN release, or not. But if it doesn't meet Toby's definition of experiment, then fine, and my apologies to Toby.
• A number of modules in the Acme namespace :-)

Furthermore, I'm not imagining that such code would be forever banished to GitHub, never allowed to sully CPAN. Once code isn't an experiment, then I see it being uploaded to CPAN as well.

My process

This "experiments on GitHub not CPAN" idea addresses just one small part of the "problems with CPAN"; I was thinking about a much smaller percentage of CPAN modules that many of you seemed to think. Mea culpa.

My personal process when creating modules will now be something like:

• Put the module on GitHub.
• Possibly register the namespace. I haven't always done this, but when I have, I've had thoughtful and helpful comments.
• Search for similar modules, and link to them in the SEE ALSO section.
• If there's an existing module close enough, see if I can contribute to that rather than release my module, or if it's gone stale, whether I can take it over.
• If it's not experimental, and I got to this point, then release to CPAN.
• Possibly write a blog post on it.

Most (if not all?) of the people who've commented on this, and related posts, are not part of the demographic I'm worrying about (read: screw the lot of you! ;-) They are: new or casual Perl programmers and CPAN users. CPAN is currently a seriously sub-optimal experience for such users.

A few years ago I was a born-again Perl newbie, and often when I turned to CPAN for "a module to do X", I'd find a handful of modules and no easy way to determine which was the right one to use. I decided I'd do a quick (ha!) review whenever I hit this, so (a) I'd make an informed decision, (b) it might help others, and (c) the peanut gallery might point out gaps / flaws in my reviews, and improve the end result. After doing a few reviews, I gave a talk on CPAN Curation at LPW 2011, where I listed some of the problems I saw with CPAN, and thoughts for how they might be addressed. I'll revisit that in a separate post, as I've ended up thinking about it a lot over the last 2 or 3 days...

The permissions model is case-sensitive, which means that Foo::Bar is treated as a different module from foo::bar, and Foo::bar. This is all fine if your filesystem is case-sensitive. But MacOSX's filesystem isn't, so this can cause problems. File::stat is a core module. I wasn't aware of File::stat's existence, and installed File::Stat (different case of 'S' on 'Stat'). In this case I got lucky: File::stat wasn't overwritten because it's a core module and File::Stat was installed in site_perl, where it got picked up first. I only discovered this because I'm using CPAN::Reporter, and various modules I tried to install reported fails to CPAN Testers, because the wrong module was being run. Fortunately for me, all I had to do was delete File::Stat from site_perl.

But this isn't the only example. I wrote a script to look for other cases, and here are some of the ones it found:

• underscore and Underscore.
• sane and Sane.
• CONFIG::Hash and Config::Hash.

There are also a lot of examples where at least one of the classes is provided by a differently named module.

• any and Any (provided by LLg).
• super (in Class::ClassDecorator) and SUPER.

These are still allocated permissions by PAUSE, so you can't define a class within another of your modules if your inner class has the same name as something already on CPAN. When you try and upload it to PAUSE, it will fail at the indexing stage. So, for SUPER and super, looking at 06perms.txt:

    SUPER,CHROMATIC,f
    SUPER,P5P,m
    super,DROLSKY,f

I also found plenty of examples where someone has changed the capitalisation of a class name between releases; for example:

    GeoHash in dist Y/YA/YAPPO/GeoHash-0.01.tar.gz
    Geohash in dist Y/YA/YAPPO/Geohash-0.04.tar.gz

This is a real problem: I broke a number of modules and only discovered it when two CPAN authors emailed me about CPAN testers reports. As a percentage of CPAN, there aren't all that many broken cases of the first type, but there are more of the second type above. And the number of CPAN dists is only increasing, so this could happen more and more.

I think PAUSE should be made case-insensitive for module permissions, and the currently 'broken' cases should be resolved (which would be a pain, I know). Another approach would be to continue letting this happen, but notify authors when they upload modules which will cause problems on MacOSX. Perhaps search.cpan.org might flag them in the same way it flags 'unauthorised releases'?

Update:

I've forked PAUSE, and am going to have a go at making this change.

Here's my current list of questions. I've got these in a Google docs form, but won't show the options for each question.

• Is Perl your primary development language?
• What level of Perl experience would you give yourself?
• How long have you been programming in Perl?
• What version(s) of Perl do your systems run on?
• Is programming in Perl part of your daily job?
• How do you find reusable code, libraries and tools?
• Have you released any code to CPAN?
• Have you written any code in Perl 6?
• What is your preferred OO framework?
• What development environment(s) do you use for Perl?
• What framework, if any, do you use for web development?
• Other than Perl, what are your primary development languages?

What other questions should be on the list?

If you program in Javascript, go fill in that survey as well.

This post outlines my next idea, and the top 20 dists based on my first implementation.

• Released is the date of the last release of the dist.
• Bug days is the number of days since the last open bug was raised.
• Gap is the number of days between the last release and the most recent bug.
• Score is Gap / bug days.

Here's the top 20 for a slightly different measure. In the table below, gap is the number of days between the most recently reported still-open bug and the oldest still-open bug. If there's only one bug, then gap will be 1, so the dist won't appear here.

Note: these are really identifying modules that are potentially worthwhile candidates for taking over (getting co-maint), rather than modules where you could contribute without having to take over maintenance. That's a separate list!

Some thoughts for improving this:

• It's skewed towards bugs raised within the last few days — too much so. Maybe instead of bug days as the denominator, I should use log10, to smooth things out.
• A dist may have been bug-free until yesterday, so hasn't needed any releases. I could look at the number of bugs that have been reported since the last release, that are still outstanding.
• Even further, I could look at the elapsed time between the oldest open bug and the most recently reported open bug.
• Factor in the number of dists that are dependent on each dist, and weight the score based on this.

What else should be factored in? I'll play a bit more, then put a longer sortable list online. I love the fact that I could get hold of all the metadata needed to create this. Now I feel like I should find a bug to fix!

Put a summary of your post here. Think of it as your pitch: you're trying to convince the reader to click on Continue reading.

Then click on the EXTENDED tab, and write the full body of your post in there.

So, why should you bother?

• It's much more considerate of the people who are reading this via a browser. If everyone did this, more posts would fit onto each page, and the reader can skim the abstracts to decide which they're interested in reading. Rather than scrolling past pages of code, looking for the next post.
• This doesn't affect anyone reading via a feed: the full article is delivered.
• It can help you with your writing. I always write the abstract first, knowing that this is all you're going to see, so I need to make it accurately and concisely summarise what follows.

I wasn't even aware of this until Shawn Corey pointed it out.

  % prereq-grapher Module::Path
  6 modules processed. Graph written to dependencies.dot

And then you can visualise this graph using graphviz:

While writing my review of CPAN modules for getting module dependency information, I knocked up a script to build the prerequisite graph using Perl::PrereqScanner. Last week someone emailed me asking if the script was available, so I put together a release...

Updates (for version 0.04)

The -nc and -nrc control whether and how core modules are included in the graph. The -nc (or -no-core) option says not to include core modules:

  % prereq-grapher -nc HTTP::Client
  2 modules processed. Graph written to dependencies.dot

Here's the output from that:

The -nrc (or -no-recurse-core) option says to include core modules, but don't include their dependencies):

  % prereq-grapher -nrc HTTP::Client
  17 modules processed. Graph written to dependencies.dot

Here's the output from that:

By default it's now silent when dependencies aren't found locally; if you include the -verbose option you'll get a warning for each dependency not found.

Update 2: limiting depth

Paul Crovella told me he was using this to look at a rather large dependency graph, so I thought it might help to be able to create a subset of the graph, by limiting the depth recursed to. I release 0.05 today, which supports a -depth option.

For example, graphing App::PrereqGrapher produces a huge graph, after processing 305 modules. Here's just a small extract of the full graph:

So, to get your head around dependencies you might then try:

  % prereq-grapher -d 2 App::PrereqGrapher
  10 modules processed. Graph written to dependencies.dot

This produces a much more useful graph:

I followed my usual sequence: openssl, apache, perl. Everything compiled with no problem. But when I installed mod_perl 2.0.7 and tried to start apache, I got the following error:

  httpd: Syntax error on line 60 of /usr/local/apache/conf/httpd.conf:
    Cannot load /usr/local/apache/modules/mod_perl.so into server:
    dlopen(/usr/local/apache/modules/mod_perl.so, 10):
    Symbol not found: _modperl_handler_name
    Referenced from: /usr/local/apache/modules/mod_perl.so Expected in: dynamic lookup

I'll spare you the gory details, but the problem was caused by the fact that there are two different C compilers, their interpretation of inline is different, and different configure processes pick up different compilers.

  % /usr/bin/cc --version | head -1
  Apple clang version 4.1 (tags/Apple/clang-421.11.66) (based on LLVM 3.1svn)

  % /usr/bin/gcc --version | head -1
  i686-apple-darwin11-llvm-gcc-4.2 (GCC) 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2336.11.00)

OpenSSL, Apache, and the apache portable runtime (APR) were all built with gcc. But Perl's Configure was picking up cc. So when building mod_perl, it was referring to an APR built with gcc, but Perl's config was telling it to use cc. Rebuilding everything and ensuring gcc was used throughout resolved the issue.