Paws XXXX (A new Start)

By byterock on December 23, 2019 9:00 PM

The next chapter in the 'Book of PAWS' is to the S3 control working.

I do not see this project to go on to long as the S3 as there are only eight actions that are exposed by the S3control API. Given this, the many fixes I have made to the XML Rests side of the perl code my full test suite and my handy test generating caller. I think I can get this control working very quicky.

Well to start out I decided to have a quick look at the Documentation for AWS S3 Control
https://docs.aws.amazon.com/AmazonS3/latest/API/API_Operations_AWS_S3_Control.html and looking there I notices there where some sixteen calls not the eight that I found when I was poking about in the boto and perl code.

It seems the documentation that is on line is for AWS S3 V2 while the Boto being used for the present PAWS and the python code is for AWS S3 V1.6 Something. I did take a look about and I notices that the latest 2 version for the Python code is still in beta version so for now I will just stick with what is in PAWs proper.

So my fist command is 'CreateJob' and I noticed something right away.

<?xml version="1.0" encoding="UTF-8"?>
<CreateJobRequest xmlns="http://awss3control.amazonaws.com/doc/2018-08-20/">
   <ConfirmationRequired>boolean</ConfirmationRequired>
   <Operation>
...

The XML body of the call is a little different that what I have seen in S3. It seems that all of the S3Control calls require that there is wrapping tag that defines the call. I better fix that before I even make my first call out to S3Control.

Well I jumped in to the boto code and for my first call I have

"input":{
    "shape":"CreateJobRequest",
    "locationName":"CreateJobRequest",
    "xmlNamespace":{"uri":"http://awss3control.amazonaws.com/doc/2018-08-20/"}
},

and looking at the typical S3 call

{"shape":"CopyObjectRequest"},

I know I can work with that 'locatioNane' as this attribute pattern is not found in S3 and appears to be on all of the S3Control calls.

Well this far into the game I know I have to go and play with the 'restxml/callargs_class.tt' and after a little playing about all I had to add in was;


+[%- IF operation.input.locationName -%]
+  has _location => (is => 'ro', isa => 'Str',  default => '[% operation.input.locationName %]');
+[%- END %]
+[%- IF operation.input.xmlNamespace %]
+  has _xmlNamespace => (is => 'ro', isa => 'Str',  default => '[% operation.input.xmlNamespace.uri %]');
+[%- END %]

which will add this to all my S3Control classes;


  has _location => (is => 'ro', isa => 'Str',  default => 'CreateJobRequest');
  has _xmlNamespace => (is => 'ro', isa => 'Str',  default => 'http://awss3control.amazonaws.com/doc/2018-08-20/');

Now it is just a quick add in to the '_to_xml_body' sub of RestXmlCaller.pm;


 }
    if ($call->can('_location')){
      my $location = $call->_location();
      if ($call->can('_xmlNamespace')){
          $xml = sprintf '<%s xmlns="%s">%s</%s>',$location,$call->_xmlNamespace(),$xml,$location;
      }
      else {
         $xml = sprintf '<%s>%s</%s>',$location,$xml,$location;
      }
    }

    return undef if ( not $xml );

    return $xml;

and that should do it.

Great my generated XML is now coming out like this;

<CreateJobRequest xmlns="http://awss3control.amazonaws.com/doc/2018-08-20/">
  <Manifest>
    <Location>
        <Etag>
…
</CreateJobRequest>

A good first start I have not even make a real request yet and I have got at least my XML coming out to the according to the document.

I gave this a whirl I and I got an 'Invalid Content' error so something is still wrong in my XML

I quickly found two places where I think my XML was not coming out correctly ;

<Spec>
  <Fields>
        <Fields>Bucket</Fields>
        <Fields>Key</Fields>
  </Fields>

which should be this;

<Spec>
  <Fields>
        <INVALID-TYPE-NAME>string</INVALID-TYPE-NAME>
        <INVALID-TYPE-NAME>string</INVALID-TYPE-NAME>
  </Fields>

and this;

<S3PutObjectTagging>
        <TagSet>
          <Key>Key 1</Key>
          <Value>value 1</Value>
        </TagSet>
        <TagSet>
          <Key>Key 2</Key>
          <Value>value 2</Value>
        </TagSet>
      </S3PutObjectTagging>

which should be this;

 <S3PutObjectTagging>
         <TagSet>
            <S3Tag>
               <Key>string</Key>
               <Value>string</Value>
            </S3Tag>
         </TagSet>
      </S3PutObjectTagging>

Hmm so something not the same as S3.

I had a look though all of the boto code for S3Control and discovered that all of the "type":"list", attributes where like this one;

"S3TagSet":{
   "type":"list",  
   "member":{"shape":"S3Tag"}
},

Unlike S3 where they either had an extra 'Flatten” attribute;

"AllowedHeaders":{
   "type":"list",
   "member":{"shape":"AllowedHeader"},
   "flattened":true
},

or a location name in the 'member' attribute;

"Buckets":{
   "type":"list",
   "member":{
      "shape":"Bucket",
      "locationName":"Bucket"
   }
},

Again something I can work with.

I started out trying to adapt the template for the RestXML calling class 'callargs_class.tt' but in the end I had to abandon that track as did get it to work for S3Contorl I broke most of S3 with the changes I made.

I concentrated instead on getting the XML generation code in RestXMlCaller.pm working as I had three types of lists to play with

Flattened
List Name In Request and
Default;

Well I did sort of get things working but I started to get this kind of XML;

  <Fields>
        <>Bucket</>
…

where I was missing the tag and looking at what I was suppose to send something called ' INVALID-TYPE-NAME' and that left me very puzzled.

Fortuntely for me I happened to stumble on the answer searching on the wayback machine. I found the API from a long defunct SOAP site that that just happened to have example requests that used such a tag. 90% of the documention of the old site was there and it included both example requests and responses.

It seem's this 'INVALID-TYPE-NAME' is just a fancy XML way of saying 'any well formed tag you want'

Some more poking about and I manages to find an old pre AWS CLI python example where they just used the word 'member' for the tag.

I tried the current AWS CLI and saw they still so this so I just did the same.

In the end I had to do was default first to the Tag name of 'Member' for all my List attributes and then change it when there was a 'flatten' or 'ListNameInRequest' trait present;

I just had to make this change in a few places;


-            my $location = $attribute->request_name;
+                       my $location = 'member';
+                       $location = $attribute->request_name
+                         if ($attribute->can('request_name'));

and now I am getting

<Spec>
  <Fields>
    <member>Bucket</member>
    <member>Key</member>
  </Fields>

I was still getting a minor problem with some of the XML as I would get an oddly invalid XML and it turns out I had an undiscovered bug from my last round of changes;



-            $xml .= "<${att_name}>" . $xml . "</${att_name}>"

+            $xml = "<${att_name}>" . $xml . "</${att_name}>"

               unless ( $attribute->does('Flatten') );

With these changes I reran all my previous tests which all pass but I am still getting a
' 'Can\'t connect to s3-control.us-east-1.amazonaws.com:443 (hostname verification failed)'

Well at least my XML is valid.