mo tag:blogs.perl.org,2009-11-03:/users/mo//437 2011-07-13T13:00:39Z mo @ #metacpan / irc.perl.org Movable Type Pro 4.38 Exciting updates to MetaCPAN tag:blogs.perl.org,2011:/users/mo//437.1968 2011-07-13T12:31:41Z 2011-07-13T13:00:39Z MetaCPAN went live with the switch over to Catalyst (previously we hadn't been using a framework at all) this morning. The main aim was to make it easier for people to contribute and also to extend. An unintended consequence was... mo http://blog.netcubed.de MetaCPAN went live with the switch over to Catalyst (previously we hadn't been using a framework at all) this morning.

The main aim was to make it easier for people to contribute and also to extend.

An unintended consequence was a significant increase in speed. We've not delved into the details - but the lesson learned is that using something as established and polished as Catalyst really makes a big difference compared to putting something together from scratch.

We've also added an experimental 'Sign In' button - have a play! You can use either Twitter, Facebook, GitHub or PAUSE to log in. Once you have authenticated against PAUSE, you will be able to edit your author's profile page via the front-end.

]]> Introducing metacpan.org - A better search for the CPAN tag:blogs.perl.org,2011:/users/mo//437.1811 2011-06-06T14:56:34Z 2011-06-06T17:41:11Z strong{font-weight:bold} em{font-style:italic;} I'm very excited to announce the front end for MetaCPAN: metacpan.org. This is what you can expect from the new interface to the CPAN: New uploads instantly availableNew uploads to the CPAN are indexed within seconds. No more... mo http://blog.netcubed.de

I'm very excited to announce the front end for MetaCPAN: metacpan.org. This is what you can expect from the new interface to the CPAN:


  • New uploads instantly available
    New uploads to the CPAN are indexed within seconds. No more waiting for new upload to appear on the search site. Check out /recent

  • Modern site layout
    The page layout leverages the extra space you have on widescreen displays. Instead of having all the information in a top-down manner, the page's layout is made up of up to three columns. Each is filled with information about the main resource.

  • Limit search to distributions or authors
    Using the distribution: syntax, you can limit your search to a given distribution. Example: distribution:Moose around. The same works for the author: tag. This functionality is embedded on various pages across MetaCPAN.

  • Search snippets preview
    Search results include text fragments which contain the query. This helps to identify the results you were looking for.

  • Search for mirrors near you
    The /mirrors page includes a map of all mirrors and allows for a for a search of mirrors close to you. Clicking on the "nearest" link will ask you browser for your current position and then sort the mirrors accordingly. You can also filter by protocol.

  • Activity indicators
    Author and release pages contain an activity graph that shows the number of uploads by a given author or a given distribution over the past 24 month. This is very useful for evaluating whether a release is well maintained or if an author stopped releasing modules.

  • Author's page includes metadata
    Each user can add additional metadata to her PAUSE account id by uploading a author-1.0.json file to her PAUSE directory. A good example is the page of DWHEELER. The page includes links to all of his online profiles and more.

  • Advanced source code view
    Code snippets in the documentation are highlighted and names of modules are linked to its documentation. POD sections are collapsed and can be expanded. See the source of Hash::Util for an example.

  • Feeling lucky?
    Generally users are only interested in the first result of a search. Clicking on the "I'm Feeling Lucky" button or hitting shift + enter will bring you directly to the first result. You can use this to add a search keyword to Firefox and others using this url: beta.metacpan.org/search?lucky=%s.

  • Open Source
    Yes, it's open source. The github repository can be found here. Patches are greatly appreciated. If you need any help to get started hacking, join us on #metacpan (irc.perl.org).

There are still a lot of new features to come (icebox) but the API as well as the front-end are now stable enough to be used by everyone else. Also, we've finally moved to a server that can hold the whole index in memory. We'd like to thank our first sponsor, speedchilli, who are generously sponsoring our hosting and providing some seriously beefy hardware:

"We are a web solutions company providing hosting and open source consultancy. We provide reliable, cost effective hosting for customers who require anything from a single dedicated server to a fully managed hosting solution. Being a business based around open source we feel it is important to give back to the community and are very happy to support the metacpan.org project." - Stewart Hunt

Please help us improve the CPAN experience

We will track searches and analyze them to optimize the search results. If you come across odd search results, such as modules that shouldn't appear because they are outdated or modules that are missing from the results, please report them to us. We will do our best to fix them.

You can also contribute directly by forking the project. Running the front-end doesn't require any heavy dependencies and is installed in less than a minute. You should also join #metacpan on irc.perl.org. It's a very active channel and new input is highly appreciated.

]]> GitHub-like activity indicator for authors and distributions tag:blogs.perl.org,2011:/users/mo//437.1730 2011-05-06T09:08:09Z 2011-05-06T08:57:18Z (Note: open the post to see the indicators. blogs.perl.org doesn't show them on the summary page) Indexing the whole CPAN to the beta version of MetaCPAN just finished. Here is one example of what you can do with the data... mo http://blog.netcubed.de (Note: open the post to see the indicators. blogs.perl.org doesn't show them on the summary page)

Indexing the whole CPAN to the beta version of MetaCPAN just finished. Here is one example of what you can do with the data available from MetaCPAN:

This is rafl's activity over the last 24 month.

And the activity of the Plack distribution.

If you want to include these indicators on your homepage, copy paste the following code:

For distributions:


<object data="http://beta.metacpan.org/activity?res=month&distribution=Plack" width="170" height="22" type="image/svg+xml" />

For authors:


<object data="http://beta.metacpan.org/activity?res=month&author=FLORA" width="170" height="22" type="image/svg+xml" />

(Note: author name must be all upper-case)

All uploads:


<object data="http://beta.metacpan.org/activity?res=month" width="170" height="22" type="image/svg+xml" />

Right now the width, height and number of month is hard-coded. Feel free to fork and fix that at https://github.com/CPAN-API/metacpan-web (relevant files are template/activity.xml and lib/MetaCPAN/Web/Controller/Activity.pm).

]]> First week of GSoC and progress made with MetaCPAN tag:blogs.perl.org,2011:/users/mo//437.1718 2011-05-03T12:59:35Z 2011-05-03T13:22:00Z As you might have heard, my MetaCPAN project has been accepted for the Google Summer of Code. If you are unfamiliar with the proposal, you can find it here. The first few weeks are called the "bonding period". In this... mo http://blog.netcubed.de As you might have heard, my MetaCPAN project has been accepted for the Google Summer of Code. If you are unfamiliar with the proposal, you can find it here.

The first few weeks are called the "bonding period". In this period the student is supposed to bond with their mentors and the community, read docs and get to know the project.

My tasks for this period are mainly to contact maintainers and author of CPAN Ratings, CPAN Testers and CPAN Vote to figure out how to include their data in the index. With CPAN Testers this was quite easy. They provide a SQLite database that includes test results for each release on the CPAN. Unfortunately we were not yet able to contact Ask, who is maintaining the CPAN Ratings page. There is a CSV file that includes some data, but it's heavily aggregated and thus not very useful for MetaCPAN.

Also, the ElasticSearch schema has been set in stone. I don't expect any significant changes to the existing schema. I will, however, add new properties to it as new data becomes available (e.g. CPAN Testers etc.). You can find the current schema here. Some examples: the latest Moose release can be found at http://api.beta.metacpan.org/release/Moose. Information about the Moose module can be found at http://api.beta.metacpan.org/module/Moose (it corresponds to the file mapping in the schema).

Next on the list is to get to know OAuth and think of a way how users can authenticate against MetaCPAN to enable services like tagging and following of authors and distributions.

If you want to join the discussion about MetaCPAN, its API and the front-end, feel free to join us on #metacpan / irc.perl.org.

]]> Improving the CPAN experience (a GSoC summer tale) tag:blogs.perl.org,2011:/users/mo//437.1556 2011-03-20T10:33:36Z 2011-03-20T09:58:35Z What will MetaCPAN offer that other services don’t? Instant availability (new uploads are indexed within a minute) Personalisation - “follow your favourites” Searchable metadata Mashup of other CPAN related services Unified (REST) API Back-end for Android/iPhone apps, command line... mo http://blog.netcubed.de What will MetaCPAN offer that other services don’t?
  • Instant availability (new uploads are indexed within a minute)
  • Personalisation - “follow your favourites”
  • Searchable metadata
  • Mashup of other CPAN related services
  • Unified (REST) API
  • Back-end for Android/iPhone apps, command line tools etc.
  • MetaCPAN.local for companies
  • Includes BackPAN as well
  • Open-source and free

Now what?

Apply for GSoC to get this thing up and running

MetaCPAN is being developed by a group of perl coders who have jobs and all kinds of stuff on their minds. This means it is hard to get the momentum up. I got very much infected by the idea of having an API to CPAN that everyone could use and a front-end that could eventually replace search.cpan.org. So I joined the MetaCPAN group and started coding. And since I’m still a student, GSoC is a great opportunity to delve even deeper into the guts of MetaCPAN and do some serious work.

Community feedback to complete proposal

In order to finish my GSoC application I want to collect as much input as possible from the community. I compiled a list of features that I feel are nice to have and will improve the experience with CPAN. Though not all of them might be feasible or even desirable.

My application will consist of two subprojects. Improving the backend and writing a state-of-the art frontend. While search.metacpan.org is nice, it doesn’t add any additional functionality to search.cpan.org. I’d like to change that and leverage the power of metacpan.

Proposed Features

Personalization

  • Follow your favorite Modules / Authors
  • Get instant notifications on updates
  • with a diff of the Changes file
  • Add discussions to modules
  • Tag modules as installed, broken, author unresponsive etc.
  • Add metadata to your own distribution (e.g. “Looking for maintainer”, deprecated etc.)
  • “CPAN of trust”

Improved search results

Currently search.cpan.org does a decent job on searching. However, it can be improved. For example it doesn’t show previews of the search results and the relevance of the returned results is sometimes questionable.

Evaluate third-party data

The following resources can be used to adjust the scoring of search results:

  • cpanvote
  • Kwalitee
  • CPAN Testers
  • CPAN Ratings

PageRank-like scoring

Using the dependency chain, one can create a graph of modules and calculate a PageRank for each module. This will greatly enhance search results since modules with a high degree of centrality will be ranked higher.

Front-end

  • A full-text search that previews the relevant segments of the document
  • Optionally limit search to a release / distribution
  • Search for exact matches in the module name (autocompletion)
  • Search for authors based on email, name and pauseid
  • Exclude results with certain dependencies (e.g. modules using Moose or XS code)
  • Keyboard navigation and shortcuts for super fast and mouse-less browsing
  • Integrate grep.cpan.me
  • Rate distributions from inside the new front-end no need to leave the page and re-login
  • and many many more features

MetaCPAN for companies

minicpan has made it easy for companies to take control over their local CPAN requirements, but they can’t search either their local minicpan, or their own internal code.

MetaCPAN.local:

  • Will be a distribution that can be installed in your company network
  • With all the features of MetaCPAN
  • Add internal company modules to the index
  • Either index the company’s minicpan or fall back to the live CPAN
  • Every front-end developed for MetaCPAN will just work for MetaCPAN.local too

Documentation

Nobody is going to use the MetaCPAN backend if there is no documentation which guides you through the basic steps of querying the metabase or setting up your own front-end.

Your Turn

I’m very excited to hear your ideas. Please don’t think too much about implementation details. Let the developer in you rest for a moment and ask youself:

  • What do I need to access CPAN more easily?
  • What information do I want to access through MetaCPAN?
  • What data is required to further improve tools like cpanm?
  • What am I missing from search.cpan.org?
  • Basically, what can MetaCPAN and its front-end do for you?
]]>