Your Personal CPAN In The Sky
Here's an idea (actually, Michael Schwern's idea): Custom CPAN-like repositories hosted in the cloud. Imagine if you could put all the CPAN modules that you depend on (including any proprietary modules of your own) into a private (or public) repository up in the cloud. You just send your tar.gz files into the cloud where they are stored and indexed so you can build, test, and install them with the standard Perl toolchain. Every time you build your application, you'll get exactly the same versions of those modules.
Now imagine this repository in the cloud has a version control system, so you can review changes and roll back your module dependencies like you do with source code. You can also branch and merge your dependencies to experiment with upgrades or alternative modules. You can search & read the documentation for all the modules in the repository (including your proprietary modules). And you can do all this through your web browser or right at the command line!
I don't expect this to appeal to everyone. Some shops have only a few deployments or just don't care if their builds are inconsistent. On the other hand, large shops may be happy using Puppet or Chef to automate their builds. But somewhere in the middle, I think there are shops that are wrestling with their dependencies and really want to solve the problem with minimum hassle. This idea is for them.
What do you think of the idea?
Would your organization (or you personally) use such a service?
How do you manage your dependencies now? Is it working for you?
Hacker, speaker, author, dad.
I like this idea, and would use such a service. Actually, I could do with it tomorrow, if you could get your skates on :-)
We've only got a small handful of machines where I have to keep perl in sync, but my mental extrapolation of what you describe would be great. I can configure one set of modules, then know then all personal machines, stage and production are in sync, apart from when we want them not to be.
And when doing releases, I could push to this service, then (a) pull to a test machine(s), (b) pull to stage and finally pull to production.
And a web interface to your service would let me see graphically view the dependency graph, and highlight where there are more recent versions of modules, then let me drill in to see the Changes for the delta.
Start a google doc and spec it out!
It's a pitty that Cory Watson has not given his YAPC::NA talk about deploying Perl web applications. As he wrote in the abstract there is "local::lib, cpanm, Dist::Zilla, carton, pinto and [...] I hate all of them". I bet one could already create a personal CPAN in the cloud with existing tools but I don't know how. Maybe we should start with simple use cases and user stories to find out what it actually missing?
I didn't know about Cory's presentation and I was at YAPC. Did it not happen? I'll get in touch with him and find out what he's got in mind.
Yes, you certainly can build a private CPAN with existing tools. IMHO, pinto is the current best of breed. But it has a lot of dependencies and it is still evolving rapidly, so creating and maintaining your own Pinto repository isn't completely trivial. And then there is just the hassle of setting up a server and managing user access. A cloud-based solution would make all that go away.
I'd love to hear more user stories. Pinto::Tutorial::Manual covers the stories that I've been through myself or were given to me by others. But I'm sure there are lots of other ways that people want to manage their dependencies. Do tell :)
I just introduced http://www.gemfury.com/ at my YAPC::Asia talk, as a list of things to possibly steal into Perl. Gemfury provides a private repo for rubygems, PyPI and NPM.
After having driven a few of the available CPAN management tools in the real world I agree with Jeff that currently Pinto is the best of breed. Cloud sounds great but at my shop most of the prod stack is behind a firewall which is not a major hurdle - rsync or some such would fill the gap to get the DarkPAN from the cloud into a dev/tst env and then mirror to prod.
I can't speak to the guts of Pinto so I don't know if it would be the ideal choice or needs lots of work to fit a cloud model. gemfury looks "neat" but I assume there is some teeth behing the shinny UI?
There are certain downsides to using a version control system to manage your dependency versions. The most important one from my experience is that it makes it difficult to have concurrent versions. At $work we have multiple versions of our software working in production simultaneously, and each one has it own dependency list. A cpan repository that has only one index at a time will prevent multiple users and a CI server from being able to build random versions on demand (Bundler and carton do not have this problem because the dependencies are versioned with the application itself).
Cory did not make it to YAPC because he had just switched jobs and now works for Twitter. The approach he was going to talk about involved building debian or rpm packages with a bundled perlbrew and local-lib (have a look at his Dist::Zilla::Plugin::Dpkg::PerlbrewStarman). This is rougly what we are using at $work now, except that we also have a private cpan repository to keep non-public third-party modules and backpan-only dependencies. We are also working on adding client-side dependency locking to CPAN.pm, hopefully this may produce something usable for a larger audience.
Ah, that is precisely what makes Pinto unique. It can have multiple indexes, so you can keep different stacks of dependencies at the same time. The version control mechanism is customized for Pinto, so you're basically versioning the individual indexes, not the whole repository itself.
You could use Pinto to stash your private modules and CPAN/BackPAN dependencies. And within the same repository, you could have "development" and "production" versions of the index. Then you can use cpanm + local::lib to fetch all the modules (from either index) and build/test/install them into a directory. Finally, you package that directory into an rpm or debian package.
I've never worked with rpms and debian packages myself, but I think that's how it would work in theory. If you give Pinto a try, I'd be eager to hear your feedback.
I also like the idea in several aspects. But freedom and authority would be the two major factor that makes me happy.
Current CPAN basically suggests people name their distribution according to some fuzzy classification rules. Top level namespaces tend to be preserved for some sort of special purposes. IMHO this kind of framework is becoming more like some sort of barrier then support. I can probably still name my testing-related work like FrogbozTester, but with some worries that somebody will just jump out and submit a bad review saying "Why using another top-level namespace instead of putting it under Test::".
When I put random personal-use stuff into a lib, I almost always named them with very short namespace for one obvious reason -- easier to type and organize. And as long as the name is fluent enough I'm comfortable with it. However my comfort zone might be incompatible with CPAN community, and I ended up not willing to upload them to CPAN (not mentioning all the tedious authoring/releasing work needs to be done should I decided to do.)
Even though I do not have numbers with me, I believe there are a lot of software developer that feels more or less the same.
gugod, CPAN is about sharing, and if you just toss random-named stuff into the pot, you wind up with a cesspool like Python's PyPi or Ruby's Gems. Their authors are frequently vain hipsters who appear to be in a contest to make the weirdest names.
Whenever somebody makes a comparison of Perl to another language, the first pro in Perl's favor is CPAN. If we lose the community professionalism for the resource it no longer becomes a selling point. I always avoid odd-named modules unless they have reached critical mass (like Moose or Dancer) because it usually reveals the author was thinking about their own interests over that of the community to which their sharing.
I think that is Kang-min's point: to use the Perl toolchain *without* having to coordinate namespaces. A custom CPAN-like repository would let you do just that. If you eventually do decide to release to the public CPAN, *then* you can worry about namespaces. But until then, you can just point cpanm at your repository in the cloud to get your stuff, no matter what it is called.
Some developers still feel that CPAN-style distributions are only useful for code that will be released to the public CPAN. I want to help dispel that belief.