October 2019 Archives

Paws XIX (The I between)

Paws S3 is coming together nicely I just have one more 'S3' todo test to look at and that is the get 'ListObjectVersions' test.

To start I fixed up the 'ListObjectVersions' test much like this



call: ListObjectVersions
service: S3
tests:
- expected: bucket
op: eq
path: Name
- expected: my
op: eq
path: Prefix
- expected: 5
op: eq
path: MaxKeys
- expected: 0
op: eq
path: IsTruncated
- expected: my-image.jp…

Paws XVIII

Well the first thing for my S3 todo fixes was to get everything back up to github for my fix for 'GetBucketLocation' once that was done I decided to tackle the rest of the 'todo' tests.

I started out with 'GetBucketPolicy' but I notices that when I set up a real test for the code I get this as the content response;


'content' => '{"Version":"2012-10-17","Id":"S3-Console-Auto-Gen-Policy-1567770368320","Statement":[{"Sid":"S3PolicyStmt-DO-NOT-MODIFY-1567770368320","Effect":"Allow","Principal":{"Service":"s3.amazonaws.com"},"Action":"s3:PutObject"…

Paws XVII (The last of the direct line for the Bourbons at least)

I finally (well a few days ago) had a response to another of my questions up on gitihub;

-->me I added in th status as you suggests. I found it under botocore as "location":"statusCode", It is used in a number of other classes


-->they

This means that we can make the builders put the trait on the appropiate attributes, which I see you already found ?

-->me

I added it in as a new trait as you suggested, not 100% sure on the name though

-->they

I think the name would better be ParamInStatus (just as a reflex of the ot…

Paw XVI

Well I though I hit the correct fix when Paws does its XML parsing and drops the 'Root' node but I found I have to apply some sort 'trait' to my class rather than an attribute so I am going to start this time with my code for 'RestXMLResponse.pm' first and then work that fix back into boto and the templates.

What I want to do is this


 if ($ret_class->can('_stream_param')) {
      $unserialized_struct = {}
    } else {
      if (not defined $content or $content eq '') {
        $unserialized_struct = {}
      } else {
        my $keep_root = 0;
…

Paws the XV (Still Some Way to Go)

In my last post I got nicely snookered by the S3 call 'GetBucketLocation' which was one of the call that had a 'todo' test.

The XML coming back from ASW is correct

<?xml version="1.0" encoding="UTF-8"?> <LocationConstraint xmlns="http://s3.amazonaws.com/doc/2006-0301/"> EU </LocationConstraint> but no matter what I tried on 'XML::Simple' I could not get it to parse just the way I want though I did get very close with this

 my $xml = XML::Simple->new(
      ForceArray    =&g…

Paws XIV (The Sun King)

Well carrying on with my quest to clean up the Paws S3 code I decided while I am waiting for my last batch of changes to be reviewed I decided to review some of the 'TODO' tests.

TODO passed: 6364, 6370, 7896, 10045, 10052, 10137

First a 'TODO' test is a stubbed in test that will most likely fail and if it does it will not effect the final Pass/Fail of test case. It is basically one of these

The first one I am going to play with is 's3-get-bucket-location.response.test.yml which looks like this

/users/byterock/2019/10/index.html

Paws XIII (The Unlucky one)

I left off in my last post with all my bits and bods to get the proper structure in my generated perl class from the Json data. Now I have to get the 'status' out of the response and then into my class.

By now I know where to start to look to in the code to apply my changes. I started in Paws::Net::RestXMLResponse' class at its 'process' sub which looks at the response 'status' and takes either the error path (>=300) or the happy path (<300). We are interested in the happy path and that takes us to the 'response_to_object' sub.

First I can see that I have the '…

Paws XII (High Noon Dog)

In my last post I was just getting the last parts of my new 'Satus' attribute in place having only one more part to create and that is a new 'trait' in '/lib/Paws/API.pm'

I was thinking af starting by coping the present package;


package Paws::API::Attribute::Trait::NameInRequest;
  use Moose::Role;
  use Moose::Util;
  Moose::Util::meta_attribute_alias('NameInRequest');
  has request_name => (is => 'ro', isa => 'Str');

and just changing the 'Request' for 'Response'. However looking at the role I see it is also adding a 'reques…

Paws XI - III (back we go)

I finally had a response to my question from this post about adding a new attribute '_status' to the auto-generated classes that Paws creates;

I'd try to avoid leaking HTTP details to the user by default. It's true that for this API call the HTTP Status is relevant to the caller, so exposing it seems legit. I'd try to limit the scope of exposing the HTTP return status by adding an attribute role like https://github.com/pplu/aws-sdk-perl/blob/master/lib/Paws/API.pm#L1 that signal the response to object routines to…

Paws X (Well not XXX)

I had my response from my question up in github.

If I am remembering correctly we are throwing an exception because at some point in time AWS has returned invalid XML on non-200 responses (the test suite tries to covers more cases). The error message was too cryptic when invalid XML came in, so we started to throw a Paws::Exception that at least gives you an indication of what went on.

Hmm ok so the test case might be valid though I do not see how that is a problem for Paws if AWS is returning inv…

Paws The IX (Not Quite X)

I am getting close to finishing off my first Paws patch.

I left off with this test failing

not ok 27 - got exception # Failed test 'got exception' # at t/11_client_exceptions.t line 104. # expecting: Paws::Exception # found: Moose::Exception::ValidationFailedForTypeConstraint (Attribute (host_id) does not pass the type constraint because: Validation failed for 'Str' with value undef at /wwwveh/lib/x86_64-linux-thread-multi/Moose/Meta/Class.pm line 275 # Moose::Meta::Class::new_object at (eval 429) line 28 # Throwable::Error::new at /home/s…

Paws The VIII (The Henry Factor)

Ok today I am going to look a the impact my changes from the last few post have had on the systems as a whole. So far I have tinkered with a few things

  1. Got my RestoreObject to work
  2. Modified the Botocore S3 json file to add in some documentation
  3. I added code to get the '="http://blogs.perl.org/users/byterock/2019/10/paws-for-fun-and-profit-the…

Paws the VII (B4 and After)

Well I left off in my last post with having 2 of the 3 tests I created failed;

ok 10210 - Got _status eq 200 from result ok 10211 - Call S3->RestoreObject from t/10_responses/s3-restore-object.3.response not ok 10212 - Got request_id eq 195AF4042CD6DF80 from result

# Failed test 'Got request_id eq 195AF4042CD6DF80 from result'
# at t/10_responses.t line 136.
# got: 'AC65B3BF8AC3AA34'
# expected: '195AF4042CD6DF80'
not ok 10213 - Got message eq Object restore is already in progress from result

# Failed test 'Go…

Paws the VI (Pardon my Error)

Now things are going good so far but I have run into a little snag with the program that is using my S3 RestoreObject code with Paws. Namely at this point my end user is trying to make the same restore request before the first restore has finished.

Right now if I capture the error with an eval {} in my call I get in the @$ is 'Conflict';

The API documentation does state that I can get as few special errors that are interesting to me;


+---------------------------------------+-------------------------------+--------+
| Message                       …

Paws the V (The Joy of Paws)

Lest see I last left off with this in my test;

not ok 10207 - Exception accessing status: Can't locate object method "status"
via package "Paws::S3::RestoreObjectOutput" at t/lib/Paws/Crawler.pm line 19.

I think I am going to fix that but first I will as a general question up on the Paws website to get a little advice in case I am going down a rabbit hole.

While I am waiting for a response this is a good opportunity to have a look at why test driven development is fun.

Right now I have that failing test an…

Paws the IV

Well I left off from my last post doing a little update to the documentation of Paws which introduced the basic layout and concept of botocore json data and it use in auto-generating code. In today's post I am going to look at an equally important part of auto-generated code and that is testing said code.

Paws has a massive test suite which is nice and good the only problem I have with it is it is 100% static data driven.

What this means you are testing against canned expected results, or as we say in the testing world

nugas quae in nugas quae sun…

Paws for Fun and Profit. The III

I left off on my last post with a little patch to do. To recap I successfully restored an S3 file that was on 'Glacier' back into 'S3' with the Paws::S3 RestoreObject command.

There was one little point and that was in the AWS S3 API documents I found this

+------+-----------------------------------------+------------+
| Name |         Description                     | Required   |
+------+-----------------------------------------+------------+
|      | Lifetime of the restored (act…

Paws for Fun and Profit. Part the Second

So today I am going to start some actual work on Paws by playing with Paws::S3. For those that do not know S3 is the Amzazon's Simple Storage System a place where we AWS types can store just about anything we want up on the cloud.

One part of S3 is the ability to change the storage state/class of an artifact. It all depends on what you are doing with the artifact. If the artifact needs to be shared across the web, like an image, you would just put it in the 'Standard' class. If you are archiving artifacts for safe keeping and there is no requirement to get them back right away y…

Paws for Fun and Profit. Part the First.

Well I think it is time I get back to blogging as the one month hiatus I originally planned has now stretched out to I think 10 or 11 months with only two little posts on the SPW back in August..

One of the positive notes from that rather depressing (at least from a Perl perspective) meet-up was the feed back I received on my one year's worth of blogging.

A few points that where raised are;


  • I should try and do more community oriented posts rather than post from my own little development bubble.

  • Go more for quality vs quantity.

  • /users/byterock/2019/10/index.html

About byterock

user-pic Long time Perl guy, a few CPAN mods allot of work on DBD::Oracle and a few YAPC presentations