Authenticating Proxy Pain
Having just finished an enormous pile of marking, but not having enough time to get on with something more substantial I thought I'd work out (again) how to use LWP::UserAgent and WWW::Mechanize behind an authenticating proxy.
Sometimes you're lucky and a proxy url of the formhttp://user:pass@my.proxy.server:8080will work nicely. No such luck for the proxy I usually use.
Anyway, here's the way to use LWP::UserAgent behind an authenticating proxy:
use LWP::UserAgent;
my $ua = LWP::UserAgent->new;
my ($user, $pass) = qw/user pass/;
$ua->proxy(['http', 'ftp', 'https'], 'http://my.proxy.server:8080/');
my $req = HTTP::Request->new('GET',"http://www.google.com");
$req->proxy_authorization_basic($user, $pass);
my $res = $ua->request($req);
print $res->content;
And here's how to do it with WWW::Mechanize:
use WWW::Mechanize;
my $mech = WWW::Mechanize->new();
my ($user, $pass, $proxy) = qw(user pass http://my.proxy.server:8080 );
$mech->credentials( $user, $pass);
$mech->proxy('http',$proxy);
my $url = 'http://whatsmyip.net/';
my $response = $mech->get($url);
print $mech->content;
->new.
Catalyst hacker, management researcher and health informatician.
It's been a long time since I opened this code, so I don't remember if the cookies were required, but I know the site I was connecting to required them. After much trial and error, I put this together:
sub new {
my $class = shift;
my $self = {};
$self->{USER} = shift || die("need username/email to login to site");
$self->{PASS} = shift || die("need password to login to site");
# creation craziness required for firewall and https support
if (defined $ENV{HTTPS_PROXY} ) {
print "Setting up proxy\n" if $debug;
my $https_proxy=$ENV{HTTPS_PROXY};
delete $ENV{HTTPS_PROXY} if ($https_proxy);
$self->{MECH} = WWW::Mechanize->new(env_proxy=>1);
$self->{MECH}->env_proxy;
$ENV{HTTPS_PROXY}=$https_proxy if ($https_proxy);
$ENV{HTTPS_PROXY_USERNAME}=$ENV{HTTP_PROXY_USER};
$ENV{HTTPS_PROXY_PASSWORD}=$ENV{HTTP_PROXY_PASS};
} else {
print "NO proxy\n" if $debug;
$self->{MECH} = WWW::Mechanize->new();
}
$self->{MECH}->agent_alias('Windows IE 6');
if (defined $opt{USEBROWSER}) {$self->{MECH}->cookie_jar(HTTP::Cookies::Netscape->new(file=>$opt{COOKIES}));}
else { $self->{MECH}->cookie_jar(HTTP::Cookies->new(file=>$opt{COOKIES}));}
bless ($self, $class);
return $self;
} # new
elwarren: Thanks for that. I think that confirms my suspicion that there's something wrong with the API that makes dealing with authenticating proxys rather inconvenient.