CPAN modules for making HTTP requests

I've published a review of CPAN modules for making HTTP requests. This covers 20 different modules from 19 distributions. I focussed on making GET and POST requests, whether the module supports https and redirects, and some other features.

As ever, I've no doubt missed a couple — let me know if you're aware of one not on the list. Also happy to hear if there are other aspects you think should be included in the comparison.


Regarding the "why" of Mojo::UserAgent in comparison to LWP:

The feature-set speaks for itself.

@tempire: is there a version for people who don't watch podcasts?

I think the biggest argument for Mojo::UserAgent is that it integrates with Test::Mojo to easily allow for testing of a Mojolicious app. I'm nearly sure that that was why it was created.

It does do lots more though, including full HTML/XML DOM parsing with CSS3 selectors and even JSON all built in. Of course these were included to help testing, but they become useful outside the Mojolicious world. Even more, the whole system is non-blocking (if used with callbacks).

I can say, at YAPC::NA Glen Hinkle was asked why, since Mojo::UA is so awesome, hasn't it been extracted from the Mojolicious system and released to the CPAN separately and he said that though it had been discussed, there are two reasons that it won't be. First, the underlying system is deeply tied to Mojolicious's event loop and other internals so it would be hard to extract. Second, the whole system is so lean that it that its not an undue burden to install the whole thing just to the UA. It installs on basic hardware in about a minute (that a design goal!) and with `HARNESS_OPTIONS=j9 cpanm Mojolicious` on good hardware it installs in about 15s! I know that at least one prominent Perl author uses M::UA often outside of any Mojolicious app.

That's a great list, thanks for sharing.

For completeness, it'd be good to see some of the event-based modules in there - personally I use Net::Async::HTTP these days, but a quick search shows these as well:

  • POE::Component::Client::HTTP
  • AnyEvent::HTTP::LWP::UserAgent
  • AnyEvent::HTTP

There's also a few special-case options such as LWPx::ParanoidAgent or HTTP::Async.

You wish for an encode_form_data function. It is already exposed as part of a LWP dependency:

perl -MHTTP::Request::Common=POST \
-e'print POST(undef, [x=>7, y=>13])->content'

Indeed, that's a great summary Joel. There are also collections (not mentioned @ YAPC) and out-of-band debugging:

Navigate though the dom using functional chains:

my @elements = $ua->get('')
->res->dom->find('#mojobar-links > a')
->grep(sub { return 1 if shift->{href} =~ /github/ })
->each(sub { say shift })


Out-of-band debugging
Monitor the web request on stderr without touching code at all:


Leave a comment

About Neil Bowers

user-pic Perl hacker since 1992.