Authenticating Proxy Pain

Having just finished an enormous pile of marking, but not having enough time to get on with something more substantial I thought I'd work out (again) how to use LWP::UserAgent and WWW::Mechanize behind an authenticating proxy.

Sometimes you're lucky and a proxy url of the form

will work nicely. No such luck for the proxy I usually use.

Anyway, here's the way to use LWP::UserAgent behind an authenticating proxy:

use LWP::UserAgent;
my $ua = LWP::UserAgent->new;
my ($user, $pass) = qw/user pass/;
$ua->proxy(['http', 'ftp', 'https'], 'http://my.proxy.server:8080/');
my $req = HTTP::Request->new('GET',"");
$req->proxy_authorization_basic($user, $pass);
my $res = $ua->request($req);
print $res->content;

And here's how to do it with WWW::Mechanize:

use WWW::Mechanize;
my $mech = WWW::Mechanize->new();
my ($user, $pass, $proxy) = qw(user pass http://my.proxy.server:8080 );
$mech->credentials( $user, $pass);
my $url = '';
my $response = $mech->get($url);
print $mech->content;

I'm not really sure how you'd go about accessing a basic auth protected page with the WWW::Mechanize method mind you. It'd also be nice if (in both cases) whether to use a proxy, and the username and password could be handled during the call to



It's been a long time since I opened this code, so I don't remember if the cookies were required, but I know the site I was connecting to required them. After much trial and error, I put this together:

sub new {
my $class = shift;
my $self = {};
$self->{USER} = shift || die("need username/email to login to site");
$self->{PASS} = shift || die("need password to login to site");

# creation craziness required for firewall and https support
if (defined $ENV{HTTPS_PROXY} ) {
print "Setting up proxy\n" if $debug;
my $https_proxy=$ENV{HTTPS_PROXY};
delete $ENV{HTTPS_PROXY} if ($https_proxy);

$self->{MECH} = WWW::Mechanize->new(env_proxy=>1);

$ENV{HTTPS_PROXY}=$https_proxy if ($https_proxy);
} else {
print "NO proxy\n" if $debug;
$self->{MECH} = WWW::Mechanize->new();

$self->{MECH}->agent_alias('Windows IE 6');

if (defined $opt{USEBROWSER}) {$self->{MECH}->cookie_jar(HTTP::Cookies::Netscape->new(file=>$opt{COOKIES}));}
else { $self->{MECH}->cookie_jar(HTTP::Cookies->new(file=>$opt{COOKIES}));}

bless ($self, $class);
return $self;
} # new

Leave a comment

About Holy Zarquon's Singing Fish

user-pic Catalyst hacker, management researcher and health informatician.