Paws XXXXIII (More fun with XML)

he first order of the day was to clean up all the debugging code I peppered across my perl trying to find an answer to my 'Failed Signature' bug.

This did take a while and in the end the changes that I am sticking with are;



if ( my $xml_body = $self->_to_xml_body($call) ) {
$request->content($xml_body);
++ $request->header( 'content-type' => 'application/xml'); #this is an XML interface so it should have this header
}


and


--    $self->sign($request);
++   if (ref($self) eq "Paws::S3Control"){ #calls out to S3Control need the Account ID
++      $self->sign($request,$call->AccountId());
++   }
++   else {
++      $self->sign($request);
++   }

return $request;

in RestXmlCaller.pm and in 'S3V4Signature.pm'



sub sign {
-- my ($self, $request) = @_;
++ my ($self, $request,$account_id) = @_;

$request->header( Date => $request->header('X-Amz-Date') // strftime( '%Y%m%dT%H%M%SZ', gmtime ) );
-- $request->header(
-- 'Host' => $self->endpoint->default_port == $self->endpoint->port
-- ? $self->endpoint->host
-- : $self->endpoint->host_port);
-- if ($self->session_token) {
++ my $url = $request->url();
++ $url =~ s/s3-control/$account_id\.s3-control/g;
++ $request->url($url);
++ $request->header(
++ 'host' => $self->endpoint->default_port == $self->endpoint->port
++ ? $account_id.".".$self->endpoint->host
: $account_id.".".$self->endpoint->host_port);

Well now that I have all the debugging code additions cleaned up it is time to take a quick review of the changes I made to get 'S3Control' 'CreateJob 'working.

First in RestXmlCaller.pm I added it the request header 'content-type' = XML and a new call out to the sign sub of 'Paws::Net::S3V4Signature' based on the S3Control class that now passes in the account_id.

In the sign sub of 'Paws::Net::S3V4Signature'' I added in the account id into the host header

At some point, more than likely after I finish off all the XML API calls, I will have to go back and re-factor the code as there really is very little difference in the two signature call roles S3V4Signature.pm and S3Signature.

I am going to hold off for now because I am not sure how many other changes have to go in to get the other XML APIs working

After all of the above I re-ran the test suite and still got 100% pass so I was able to move forward and tackel the next problem this error;

<Code>InvalidRequest</Code><Message>Missing role arn</Message>

I first checked the XML that is I was sending vs the AWS documentation and I confirmed it is missing from the XML I was generating but I did include it in the 'parameters' that where being passed along.

So I checked Boto to make sure I did not overlook anything new



"RoleArn":{
"shape":"IAMRoleArn ",
"documentation":"<p>The Amazon Resource Name (ARN) for the Identity and Access Management (IAM) Role that batch operations will use to execute this job's operation on each object in the manifest.</p>"
}

No nothing there I haven't seen before. It simply points to a string.


"IAMRoleArn":{
"type":"string",
"max":2048,
"min":1
}, 

So the problem is not in the templates or the auto-generated classes it must be in the XML generation subs of ' RestXmlCaller.pm'

So I dove into the XML subs string and found after a little playing around there is a fall though for all the top level attributes that do not have a trait. For example in the 'CreateJob' class attributes like this one;


 has RoleArn => (is => 'ro', isa => 'Str' , required => 1);

will fall though the XML sub and not get added to the XML so I fixed that with this;



else {
$xml .= sprintf '<%s>%s</%s>',$attribute->name,$attribute_value,$attribute->name;
}

but that was a problem as there where other attributes on the class like these;



has _location => (is => 'ro', isa => 'Str', default => 'CreateJobRequest');
has _xmlNamespace => (is => 'ro', isa => 'Str', default => 'http://awss3control.amazonaws.com/doc/2018-08-20/'

and these non XML attrubtes cause invalid XML errors like this;


<_location>CreateJobRequest</_location><_xmlNamespace>http://awss3control.amazonaws.com/doc/2018-08-20/</_xmlNamespace></CreateJobRequest>

so I will have to get rid of that. The best way to acomplish this is with yet another trait. With this new trait I can check is see if an attribute is part of the API.

After playing about a little I found it would be easier to mark the few attributes I do not want then all the ones I want. So with this new trait;


package Paws::API::Attribute::Trait::isLocal;
  use Moose::Role;
  use Moose::Util;
  Moose::Util::meta_attribute_alias('isLocal');

I added it to the templates


[%- IF operation.input.locationName -%]
  has _location => (is => 'ro', isa => 'Str',  default => '[% operation.input.locationName %]', traits => ['isLocal']);
[%- END %]
[%- IF operation.input.xmlNamespace %]
  has _xmlNamespace => (is => 'ro', isa => 'Str',  default => '[% operation.input.xmlNamespace.uri %]',traits => ['isLocal'] );
[%- END %]

and at the end of my XML subs I just add in;


--else {
elsif (!$attribute->does('Paws::API::Attribute::Trait::IsLocal')) {

Now a quick recompile and I have removed those nasty extra XML attributes.

Alas the story is not over yet. Once I was done I re-ran my test suite and got some 25 errors. Seems there are Quite a few calls in S3 that are effected by the above change as the XML no longer matches up. I know have extra XML nodes popping up all over the place.

I was thinking I could just change the tests to match the new output but before I tried that I figured it was a good idea give one of my real world tests a go to make sure the S3 API would allow the extra Nodes.

It didn't

so back into my code.

Well I know that only the s3Contol class have the '_location' at the class level so maybe a different tack is need here.

I played about for a while and just create a temp var $xml_extra that I just use in the fall though and then tack onto the end like this


sub _to_xml_body {
    my ( $self, $call ) = @_;
    my $xml = '';
    my $xml_extra = '';
…

}
elsif (!$attribute->does('Paws::API::Attribute::Trait::IsLocal')) {
$xml_extra .= sprintf '<%s>%s</%s>',$attribute->name,$attribute_value,$attribute->name;
}

if ($call->can('_location')){
my $location = $call->_location();
$xml .= $xml_extra;



and all my test are passing both for S3Contgrol and S3.

Time to move on again.

NewYear.jpg

Leave a comment

About byterock

user-pic Long time Perl guy, a few CPAN mods allot of work on DBD::Oracle and a few YAPC presentations