byterock

Paging PAWS for fun an profit

Egad have been away for a while, it is not due to laziness on my part, I really have been stuck on a Paws problem over the past month+, add to that dozens of inside and out sided projects that I need to get done around the house time has just not been there.

At least I have finally cracked it.

I really went down a rabbit hole for this one and spend way too many hours trying to figure out how to test 'Paws Pagination' end to end.

In my last post I started out with a new test suite '30_pagination.t' and a few test YAMLs.

Just getting the YAML just right took God only knows how many iterations. I also had to create a completely new caller 'TestPaginationCaller.pm' to get the tests to work.

So here is the 25c story on how it works. I start with the normal two YAML files, one for content and one for tests. The differance is that I have both the 'request' and 'response' content and tests in each. The content file looks like this;

---
requests:
  - Limit: 2
  - Limit: 2
    ExclusiveStartStreamName: Test1
…
  - Limit: 2
    ExclusiveStartStreamName: Test5
responses:
  - content: "{\"HasMoreStreams\":true,\"StreamNames\":[\"Test1DataStream\",\"Test1\"]}"
    headers:
      content-length: 65
      content-type: application/x-amz-json-1.1
    status: 200
  - content: "{\"HasMoreStreams\":true,\"StreamNames\":[\"Test2DataStream\",\"Test2\"]}"
    headers:
      content-length: 72
      content-type: application/x-amz-json-1.1
    status: 200
…
  - content: "{\"HasMoreStreams\":false,\"StreamNames\":[\"Test6DataStream\"]}"
    headers:
      content-length: 53
      content-type: application/x-amz-json-1.1
    status: 200

while my test YAML looks like this;

---
call: ListAllStreams
service: Kinesis
request:
  pages:
    - tests:
       - path: content
         expected: "{\"Limit\":2}"
         op: eq
         type: json
       - path: headers
         key: content-type
         expected: "application/x-amz-json-1.1"
         op: eq
       - path: method
         expected: POST
         op: eq
       - path: parameters
         key: Action
         expected: ListStreams
         op: eq
…
response:
  pages:
    - tests:
       - type: ARRAY
         expected:
           - Test1DataStream
             Test1
...

To run tests tests I have, '30_pagination.t'. I won't put all of that code here, suffice it to say I just load in the content and tests and then iterate over each of the 'pages' and run the tests.

Sounds simple enough, but here is the rub.

When I set up the call to 'ListAllStreams' with the added in pagination sub like this;

my @shards =();
my $count =0;
my $ListOutput = $s3->ListAllStreams(
     sub { push(@shards,$_); $count++;
     },
    Limit  => 2
);

I ran into an endless loop; The reason for this was in the auto-generated Kinesis.pm code;

 if (not defined $callback) {
…
    } else {
      while ($result->HasMoreStreams) {
        $callback->($_ => 'StreamNames') foreach (@{ $result->StreamNames });
        $result = $self->ListStreams(@_, ExclusiveStartStreamName => $result->StreamNames->[-1]);
      }
      $callback->($_ => 'StreamNames') foreach (@{ $result->StreamNames });
    }

You can see there is a while loop that will never be broken without changing the Kinesis.pm file which I was not about ready to do.

So there I am thinking it would it would be a rather easy task, just set up the call and test the output. Sister! Was I ever wrong!

I tried every Perl Monk incantation on this one with little or no luck until I finally figured out that at I could at least pass a call to a local sub in that 'callback'. I tried that like this;

 $result = $service->$call_method(
                    sub {
                        _test_result($_ );
                    },
                    %{$request}
                )

Now I had at least something to play with. Well I eventually broke out of the endless loop but only by using a 'die' which was not really a long term solution.

With the 'Callback' sub calling another sub I eventually was able to test the content from the response. I had to add in a few new attributes to my 'TestPaginationCallerpm' to get what I need in the sub. In the end my call looked like this;

   _test_result($service->caller, $service->caller->response_test->tests->[0], $_ );

and the sub like this;

 sub _test_result {
        my ( $caller,$test, $result ) = @_;
        my $count = $caller->counter();
        my $max   = scalar(@{$test->{expected}})-1;
        my $expected = $test->{expected}->[$count];
        ok( $expected eq $result,
            "Got expected result for #$count $expected=$result" );
        if ( $count == $max ) {
            $caller->reset_counter();
            die;
        }
        $caller->inc_counter();
    }

I just let the while loop run until my internal counter is reached then die. It worked, but not a solution;

I had but one option left, something I have never done in Perl. As a mater of fact the last time I did this I was playing with Applesoft BASIC on a Unitron 2200.

I give now fair warning to any sensitive readers out there to stop here and go read something else.

Please please do not look at the next few lines of code it may cause gnashing to teeth and pulling of hair;

You have been warned!!

Last chance!!

You will regret it!!!

Think of the children!!!

I solved my problem like this;

$result = $service->$call_method(
                    sub {
                        _test_result($_ );
                    },
                    %{$request}
                )
 NEXT_REQUEST:
…

     if ( $count == $max ) {

        $caller->reset_counter();

        goto NEXT_REQUEST;^

…

Actually after I go the above squared away the rest of the changes where just simple cut and paste job to test the 'requests' while in the 'do_call' sub 'TestPaginationCallerpm.pm' rather than in the t/30_pagination.t. file. Hence the Test on the first part if its name.

Finally back on track and hopefully my next post will not take as long.