Paws XXXXI (Doubble Sawbuck plus 1)

I was hoping that I would have an easy time with S3Control once I figured out the XML problems. Sadly I was a little premature on that thought.

I spent a good while over the past few days getting either an 'Access Denied' or 'Forbidden' response to my 'CreateJob' call.

I eventually though that I might as well try one of the simpler calls and get that working and then work may way up. So I tried the simplest of all;

 my $s3 = Paws->service(
    region => 'us-east-1',
    debug  => 1,
    caller => FullTestMakerLWPCaller->new(
         make_response_test => 1,
         make_request_test  => 1,
         warn_response      => 1,
        warn_request       => 1,
        response_test_dir  => '/home/scolesj/S3Control',
        request_test_dir   => '/home/scolesj/S3Control'

my $call = $s3->ListJobs(AccountId => '985173205561');

The ' ListJobs' and I was still getting the same 'Forbidden' results. I even added a job directly on AWS so I would get at least some result.

Well I eventuality tried making my call with the python CLI and then I found it. The call it was making was;;
compared to what I was sending
I checked the AWS API Documentation
GET /v20180820/jobs?JobStatuses=JobStatuses& MaxResults=MaxResults&NextToken=NextToken HTTP/1.1 x-amz-account-id: AccountId

hmm no indication in the above that one needs to add in my account ID at the front of the URL like that.

So I hard-coded that in and on the next run I got;

 bless( {
'_request_id' => '4877041D541E6554',
'Jobs' => [ <ListJobsResult>


at least in the Raw format or the response so it worked.

So time to see what is going on.

I searched high and low in the Boto code so something that might trigger this but alas to no avail. I finally got desperate enough to look at the Python code for the CLI and I found this;

      # S3 Control

Turns out this is a one off call for S3control that adds in the Account id at the beginning of a URL before the call is made at least that is what the comment on the Python code says

  Hoist a header to the beginning of the hostname with a suffix "." after
        it. The original header should be removed from the header map. This
        method is intended to be used as a target for the before-call event.

This python code is some 30 lines long and 3 calls to various subs so I am not going to include it here. This is the perl code I used to acomplish the same result;

  my $url = $self->_api_endpoint . $uri;
  if (ref($self) eq "Paws::S3Control"){
       my $account_id = $call->AccountId();
     $url =~ s/s3-control/$account_id\.s3-control/g;


and I get a URL like this
But I was still out of luck as now I am getting a “Forbidden” Drats and Darns!

After many many run I noticed this

 'host' => '',

on the request object. It should be;

'host' => '',

This was getting a little too low level as I found this comment in the code;

   # HTTP::Tiny has made setting Host header illegal. It derives Host from URL
    delete $headers->{Host};
At first I though my HTTP::Tiny might have been out of date as it was only version 0.47 but I updated it but that did not help.

I even went down and hacked up ' HTTP::Tiny', event though this is very unwise, I did not find any problem in there.

I did hard code the host and rather getting a 'Forbidden' I started getting a 'SignatureDoesNotMatch' error well at least that was something different.

That lead me to the last call in the 'prepare_request_for_call' sub in;

    $self->_to_header_params( $request, $call );

So the 'sign' call is out of wack so I looked in there next and that brought me into 'Paws::Net::S3V4Signature' role. Now I am getting into a very low level indeed.

Looking in that I quickly dicovered that S3Control is the only PAWS mod that uses it so I think I have 100% free reign in there to play about.

The first thing I noticed is that it uses the 'endpoint' attribute of the Caller and looking back I see that this is a lazy call that links all the way back to the '_construct_endpoint' sub found in

Paws::API::EndpointResolver class.

Now the only problem with that 'Role' is that is is used in 100% of the calling classes so I going to have to be very careful playing in here.

Well looking at the attribute I want to play with;

has endpoint => (
    is => 'ro',
    isa => 'Paws::EndpointURL',
    lazy => 1,
    coerce => 1,
    default => sub {
      shift->_endpoint_info->{ url };

I see that is is read-only and fires when it is called in the 'BUILD' call of 'S3V4Signature'. Looking deeper in there and playing about a bit it quickly became clear to me that I will not be able to change any of the code in the 'Paws::API::EndpointResolver' as it links backs to hard-coded attribute subs like this;

sub service { 'signin' }

found in the Paws::Signin class.

Also attempting to play about with the BUILD sub of 'S3V4Signature' was a no go as well as that code will fire long before I get any of the calls params so I will not have the AWS 'Account ID' at this point.

So that left me with only option of making my changes in the 'sign' sub of 'S3V4Signature'.

Well enable this I first I will have check to see if I am a "Paws::S3Control" before I do the sign like this at the end of the 'prepare_request_for_call' in

if (ref($self) eq "Paws::S3Control"){
    else {

and on the 'S3V4Signature' I just need to patch it like this;

 sub sign {
-    my ($self, $request) = @_;
+    my ($self, $request,$account_id) = @_;

$request->header( Date => $request->header('X-Amz-Date') // strftime( '%Y%m%dT%H%M%SZ', gmtime ) );
+ my $url = $request->url();
+ $url =~ s/s3-control/$account_id\.s3-control/g;
+ $request->url($url);
'Host' => $self->endpoint->default_port == $self->endpoint->port
- ? $self->endpoint->host
- : $self->endpoint->host_port);
+ ? $account_id.".".$self->endpoint->host
+ : $account_id.".".$self->endpoint->host_port);
if ($self->session_token) {
$request->header( 'X-Amz-Security-Token' => $self->session_token );

and presto I am getting

 bless( {
 '_request_id' => '4877041D541E6554',
   'Jobs' => [ <ListJobsResult>


Well that was a long one.

I guess I will have to check and see what I might have broken?


1 Comment

The python/perl meme would've been better if the line hadn't come from a character who's literally an irredeemable asshole.

Leave a comment

About byterock

user-pic Long time Perl guy, a few CPAN mods allot of work on DBD::Oracle and a few YAPC presentations