byterock

I was hoping that I would have an easy time with S3Control once I figured out the XML problems. Sadly I was a little premature on that thought.

I spent a good while over the past few days getting either an 'Access Denied' or 'Forbidden' response to my 'CreateJob' call.

I eventually though that I might as well try one of the simpler calls and get that working and then work may way up. So I tried the simplest of all;

 my $s3 = Paws->service(
    'S3Control',
    region => 'us-east-1',
    debug  => 1,
    caller => FullTestMakerLWPCaller->new(
         make_response_test => 1,
         make_request_test  => 1,
         warn_response      => 1,
        warn_request       => 1,
        response_test_dir  => '/home/scolesj/S3Control',
        request_test_dir   => '/home/scolesj/S3Control'
    ));

my $call = $s3->ListJobs(AccountId => '985173205561');

The ' ListJobs' and I was still getting the same 'Forbidden' results. I even added a job directly on AWS so I would get at least some result.

Well I eventuality tried making my call with the python CLI and then I found it. The call it was making was;;

https://985173205561.s3-control.us-east-1.amazonaws.com/v20180820
/jobs

compared to what I was sending

https://s3-control.us-east-1.amazonaws.com/v20180820/jobs

I checked the AWS API Documentation

GET /v20180820/jobs?JobStatuses=JobStatuses&
MaxResults=MaxResults&NextToken=NextToken HTTP/1.1
x-amz-account-id: AccountId

hmm no indication in the above that one needs to add in my account ID at the front of the URL like that.

So I hard-coded that in and on the next run I got;

 bless( {
'_request_id' => '4877041D541E6554',
'Jobs' => [ <ListJobsResult>
  <NextToken>string</NextToken>
  <Jobs>
  <JobListDescriptor>

...

]

at least in the Raw format or the response so it worked.

So time to see what is going on.

I searched high and low in the Boto code so something that might trigger this but alas to no avail. I finally got desperate enough to look at the Python code for the CLI and I found this;

      #############
      # S3 Control
      #############
      ('before-call.s3-control.*',
       HeaderToHostHoister('x-amz-account-id').hoist),
  

HA!

Turns out this is a one off call for S3control that adds in the Account id at the beginning of a URL before the call is made at least that is what the comment on the Python code says

  Hoist a header to the beginning of the hostname with a suffix "." after
        it. The original header should be removed from the header map. This
        method is intended to be used as a target for the before-call event.

This python code is some 30 lines long and 3 calls to various subs so I am not going to include it here. This is the perl code I used to acomplish the same result;

  …
  my $url = $self->_api_endpoint . $uri;
  if (ref($self) eq "Paws::S3Control"){
       my $account_id = $call->AccountId();
     $url =~ s/s3-control/$account_id\.s3-control/g;
    
  }

and I get a URL like this

https://985173205561.s3-control.us-east-1.amazonaws.com/v20180820/jobs

But I was still out of luck as now I am getting a “Forbidden” Drats and Darns!

After many many run I noticed this

 'host' => 's3-control.us-east-1.amazonaws.com',

on the request object. It should be;

'host' => '985173205561.s3-control.us-east-1.amazonaws.com',

This was getting a little too low level as I found this comment in the code;

   # HTTP::Tiny has made setting Host header illegal. It derives Host from URL
    delete $headers->{Host};

At first I though my HTTP::Tiny might have been out of date as it was only version 0.47 but I updated it but that did not help.

I even went down and hacked up ' HTTP::Tiny', event though this is very unwise, I did not find any problem in there.

I did hard code the host and rather getting a 'Forbidden' I started getting a 'SignatureDoesNotMatch' error well at least that was something different.

That lead me to the last call in the 'prepare_request_for_call' sub in RestXmlCaller.pm;

    $self->_to_header_params( $request, $call );
    $self->sign($request);

So the 'sign' call is out of wack so I looked in there next and that brought me into 'Paws::Net::S3V4Signature' role. Now I am getting into a very low level indeed.

Looking in that I quickly dicovered that S3Control is the only PAWS mod that uses it so I think I have 100% free reign in there to play about.

The first thing I noticed is that it uses the 'endpoint' attribute of the Caller and looking back I see that this is a lazy call that links all the way back to the '_construct_endpoint' sub found in

Paws::API::EndpointResolver class.

Now the only problem with that 'Role' is that is is used in 100% of the calling classes so I going to have to be very careful playing in here.

Well looking at the attribute I want to play with;

has endpoint => (
    is => 'ro',
    isa => 'Paws::EndpointURL',
    lazy => 1,
    coerce => 1,
    default => sub {
      shift->_endpoint_info->{ url };
    }
  );

I see that is is read-only and fires when it is called in the 'BUILD' call of 'S3V4Signature'. Looking deeper in there and playing about a bit it quickly became clear to me that I will not be able to change any of the code in the 'Paws::API::EndpointResolver' as it links backs to hard-coded attribute subs like this;

sub service { 'signin' }

found in the Paws::Signin class.

Also attempting to play about with the BUILD sub of 'S3V4Signature' was a no go as well as that code will fire long before I get any of the calls params so I will not have the AWS 'Account ID' at this point.

So that left me with only option of making my changes in the 'sign' sub of 'S3V4Signature'.

Well enable this I first I will have check to see if I am a "Paws::S3Control" before I do the sign like this at the end of the 'prepare_request_for_call' in RestXmlCaller.pm

if (ref($self) eq "Paws::S3Control"){
       $self->sign($request,$call->AccountId());
    }
    else {
       $self->sign($request);
    }

and on the 'S3V4Signature' I just need to patch it like this;

 sub sign {
-    my ($self, $request) = @_;
+    my ($self, $request,$account_id) = @_;

     $request->header( Date => $request->header('X-Amz-Date') // strftime( '%Y%m%dT%H%M%SZ', gmtime ) );

+    my $url = $request->url();

+    $url =~ s/s3-control/$account_id\.s3-control/g;

+  $request->url($url);

     $request->header(

         'Host' => $self->endpoint->default_port == $self->endpoint->port

-        ? $self->endpoint->host

-        : $self->endpoint->host_port);

+        ? $account_id.".".$self->endpoint->host

+        : $account_id.".".$self->endpoint->host_port);

     if ($self->session_token) {

       $request->header( 'X-Amz-Security-Token' => $self->session_token );

     }

 

and presto I am getting

 bless( {
 '_request_id' => '4877041D541E6554',
   'Jobs' => [ <ListJobsResult>
   <NextToken>string</NextToken>
   <Jobs>
      <JobListDescriptor>

...

]

Well that was a long one.

I guess I will have to check and see what I might have broken?