byterock

Well I left off from my last post doing a little update to the documentation of Paws which introduced the basic layout and concept of botocore json data and it use in auto-generating code. In today's post I am going to look at an equally important part of auto-generated code and that is testing said code.

Paws has a massive test suite which is nice and good the only problem I have with it is it is 100% static data driven.

What this means you are testing against canned expected results, or as we say in the testing world

nugas quae in nugas quae sunt

or in other words you are not really testing anything you are just testing a test.
Lets look at a typical Paws test, the tests for S3 DeleteObject
It consists of two Yaml files the 'test';

call: DeleteObject
service: S3
tests:
  - path: _request_id
    op: eq
    expected: 30EA1A44D6905562

and the data file

---
content: ~
headers:
  connection: close
  date: 'Tue, 01 Nov 2016 21:58:15 GMT'
  server: AmazonS3
  x-amz-id-2: FQbAr3B+YeF0M2B918n1e8aosHkw0gQ8zh3v2HOm5FL0KRuKopQ/2764w4ZVzJJnWFahVHjU6tM=
  x-amz-request-id: 30EA1A44D6905562
status: 204

This test is run as part of the '10_responses.t' case. The code, when run, creates an instance of the “DeleteObjectOutput” class using the 'data' file and then tests the '_request_id' attribute of the instance against the value of 'x-amx-request-id' in the test 'data'.

I am not bashing this type of testing as it is does prove the data being parsed into the 'DeleteObjectOutput' is correct. The only fault is one is not really testing against AWS, but then again to test all of the code against a live AWS instance would be impractical.

First every end user would be required to have a AWS account and ensure that the account has all the correct permissions, privileges, and the correct canned data are present.

Secondly some AWS actions cost money to invoke. Some like the 'S3' 'RestoreObject,' from my last post, may only cost a fraction of a cent but some like EC2 'CreatetInstance' could cost you a few dollars to run.

Anyway long story short the best we can do is test against know canned results.
To get a little back on track what I have to add is a test for S3 'RestoreObject' as it is missing from the present test suite. The class I will be testing is the 'Paws::New::APIResponse'. To start off all I to do is make a copy of the 'DeleteObject' test found in the t/10_responses dir.

cp s3-delete-object.1.response s3-restore-object.response

cp s3-delete-object.1.response.test.yml s3-restore-object.response.test.yml

and now I have something to play with.

Next I have a look at the successful response I had back in this post.

bless( {
                 'content' => '',
                 'headers' => {
                                'server' => 'AmazonS3',
                                'date' => 'Fri, 20 Sep 2019 23:18:30 GMT',
                                'content-length' => '0',
                                'x-amz-version-id' => 'null',
                                'x-amz-id-2' => 'F/Aqv8iKKyDxL9zJdqkDtm4iBgKtL2B+AXlEiNE7HzjwcDaLroGM7duOZTCCIYT/ITxFqlu3teU=',
                                'x-amz-request-id' => '9C4494F17CFA569D'
                              },
                 'status' => '200'
               }, 'Paws::Net::APIResponse' );

and make some changes to the s3-restore-object.response file

---
content: ~
headers:
  connection: close
  date: 's3-restore-object.response'
  server: AmazonS3
  x-amz-id-2: F/Aqv8iKKyDxL9zJdqkDtm4iBgKtL2B+AXlEiNE7HzjwcDaLroGM7duOZTCCIYT/ITxFqlu3teU=
  x-amz-request-id: 9C4494F17CFA569D
status: 200

to match up with my real world test from the other day.
Next I fix up the s3-restore-object.response.test.yml file to check the correct class and the expected results;

---
call: RestoreObject
service: S3
tests:
  - path: _request_id
    op: eq
    expected: 9C4494F17CFA569D
-object.1.

And I should be good to go.
When I run the test suite with

prove   t/10_responses.t

I get;

...
ok 10205 - Call S3->RestoreObject from t/10_responses/s3-restore-object.response
ok 10206 - Got _request_id eq 9C4494F17CFA569D from result
ok 10207 - Call S3->UploadPart from t/10_responses/s3-upload-part.response
…

To make sure I got everything right I had a look at the API docs;

Responses

• A successful operation returns either the 200 OK or 202 Accepted status code.
• If the object copy is not previously restored, then Amazon S3 returns 202 Accepted in the response.
• If the object copy is previously restored, Amazon S3 returns 200 OK in the response.
• Response Headers
• This implementation of the operation uses only response headers that are common to most responses. For more information, see Common Response Headers.

Interesting there are two paths for the 'status' 200 and 202. So lets modify our test a little to check that status.

- path: _request_id

    op: eq

    expected: 9C4494F17CFA569D

- path: status

    op: eq

    expected: 200

and I rerun my tests and get;

not ok 10207 - Exception accessing status: Can't locate object method 

"status" via package "Paws::S3::RestoreObjectOutput" 

at t/lib/Paws/Crawler.pm line 19.

Ok something is amiss there as I think I would want to know if I am getting s 202 or or 202 back as my status.

Oh well that is another story.