Back to Paws

By byterock on March 25, 2020 9:00 PM

It has been a little while since I played with my little PAWS and yes like many of us these days I have been just a little distracted, trip planned, trip changed, trip canceled etc etc etc.

Anyway to recap where I left off I was just getting the 'SubscribeToShard' action to work with a HTTP stream to work, after a fashion anyway. Then I got side tracked a little playing about with the problem of testing if the stream was correctly sending data down the pipe and if I was decoding it correctly.

As a byproduct of getting to the bottom of that I finally figured out what the PAWS 'Paginators' are for and I guess how to use them.

I noticed the odd "NextToken" tag in some of the Boto Json files as well most of the services have a ''paginators-1.json' definition file as well and looking at the Kinesis pod I see that there paginators listed.

PAGINATORS
Paginator methods are helpers that repetitively call methods that return partial results

…

Well I had to look into this as it is something new form me as I had never come across in my PAWS journey. Poking about I did find that the paginators are being generated by PAWS but there are very few test cases only 4 for the S3.

To get the paginators to work all one has to do is pass in an antonymous sub when calling the action. Below you can see this in its simplest form;


my @pages;
my $count = 0;
my $ListOutput = $kinisis->ListAllStreamConsumers(
     sub { my $page = $_;
             push(@pages,$page); 
            print("/nPage: $count");
            $count++;
    },
    StreamARN               => 'arn:aws:kinesis:us-east-1:985173205561:stream/TestSteam5Shard',
    MaxResults => 1
);

print “/nMy page count was $count”

print “/nI got ”.Dumper(\@pages);

When run the output looked like this;


Page: 1
Page: 2
Page: 3
My page count was 3;
I got [ bless( {
                 'ConsumerName' => 'TestKinesisApp',
                 'ConsumerARN' => 'arn:aws:kinesis:us-east-1:985173205561:stream/TestSteam5Shard/consumer/TestKinesisApp:1581111187',
                 'ConsumerCreationTimestamp' => '1581111187',
                 'ConsumerStatus' => 'ACTIVE'
               }, 'Paws::Kinesis::Consumer' 
…]

So it looks like I have another void to fill. There is a generic test rig for paginators present and at first glace it looks much the same as the rig for Requests and Responses but alas no automatic test generator.

Well it seems I have learned at least one thing on my PAWS journey and that is get the test squared away first. So keeping that in mind the plan for today is to come up with another caller/test generator that will create test for the paginator So like my other PAWS adventures I can write new real world test scripts and create the canned test from it at the same time.

Now where to start?? I guess 't/26_paginators.t'. Well I had a look in there and I could see it was only ½ made up with the service and action hard coded in place. Not much to start with.

Unfortunately I will not be able to reuse the 09 and 10 test suites because for a full test I have to provide both the 'Resquest', to test the anonymous sub hookup and the 'Respose' to see how PAWS handles the 'sub' part of the response which will do the iteration.

So a new test suite is required and I am going to call it 30_pagination for now.

I also had to do a deep dive into the guts of PAWS and to how it does pagination so I could get a handle on how to test it properly. The patter is the same throughout PAWs each of the paginators are a sub found in the class of the Action.

I am presently working on 'ListAllStreams' and if I go into the “Paws::Kinesis” class there is the following implementation of the sub;


 sub ListAllStreams {
    my $self = shift;

    my $callback = shift @_ if (ref($_[0]) eq 'CODE');

    my $result = $self->ListStreams(@_);

    my $next_result = $result;

    if (not defined $callback) {

      while ($next_result->HasMoreStreams) {

        $next_result = $self->ListStreams(@_, ExclusiveStartStreamName => $next_result->StreamNames->[-1]);

        push @{ $result->StreamNames }, @{ $next_result->StreamNames };

      }

      return $result;

    } else {

      while ($result->HasMoreStreams) {

        $callback->($_ => 'StreamNames') foreach (@{ $result->StreamNames });

        $result = $self->ListStreams(@_, ExclusiveStartStreamName => $result->StreamNames->[-1]);

      }

      $callback->($_ => 'StreamNames') foreach (@{ $result->StreamNames });

    }

    return undef

  }

As you can see it follows one of two paths. No callback, which I do not care about and Callback. There is not much in there for the callback. It just takes the current result, which in this case is a 'Paws::Kinesis::ListStreamsOutput' that has already been coerced from the query, iterates over each of the values in the 'StreamNames' attribute and passes each those off to the 'passed in sub' then it gets the next result and repeats till nothing is left.

As the above is template generated code, there is not much I can do here on the testing side so I am really just limited to looking at the output which in this case is just an array ref of 'Stream Names'.

Hmm.

Well the end result could be almost anything you want but in the simplest form with a call sub like this;


   sub { push(@shards,$_)},

you would get just something like this


[
          'Test2DataStream',
…
          'TestStream11',]

So I started by creating the two standard tests YAML files. I have the .test files

---
call: ListAllStreams
service: Kinesis
tests:
  pages: 5
  items: 10
  pages: 
    - expected: ARRAY
      op: eq
      values: 
         - Test2DataStream
         - Test3
    - expected: ARRAY
      op: eq
      values: 
       - Test2DataStream
       - Test3
    - expected: ARRAY
      op: eq
      values:  
       - TestSteam5Shard
       - TestStream10
    - expected: ARRAY
      op: eq
      values:  
       - TestStream11
       - TestStream2
    - expected: ARRAY
      op: eq
      values: 
       - TestStream5
       - TestStream6
    - expected: ARRAY
      op: eq
      values: 
       - TestStream8
       - TestStream9
    - expected: ARRAY
      op: eq
      values: 
       - TestStrem4

and

---
request:
  params:
    MaxResults: 1
    NextToken: mynext
    StreamCreationTimestamp: 1581111187
response:
  pages: 
    - {"HasMoreStreams":true,"StreamNames":["Test2DataStream","Test3"]}
    - {"HasMoreStreams":true,"StreamNames":["TestSteam5Shard","TestStream10"]}
    - {"HasMoreStreams":true,"StreamNames":["TestStream11","TestStream2"]}
    - {"HasMoreStreams":true,"StreamNames":["TestStream5","TestStream6"]}
    - {"HasMoreStreams":true,"StreamNames":["TestStream8","TestStream9"]}
    - {"HasMoreStreams":false,"StreamNames":["TestStrem4"]}

for the content file.

I did look about in the various pages and they all seem to return everying from simple scalars, arrays of scalars, single classes and collection of classes. So to account for this in the 'expected' I can use values ARRAY, HASH or even the Class name or perhaps even something like ARRAY[CLASS].

Well I guess that is my next post is to come up with a test for the above.