Egad not more PAWs posts :(

By byterock on February 13, 2020 9:00 PM

Well back on my PAWS run again. This one might be a rather short series as I am really just looking at one Action in the Kinesis API 'SubscribeToShard'. There is an open bug for this one up on github https://github.com/pplu/aws-sdk-perl/issues/371 and one I think I can fix up fairly eaisy.

First things first, a little word on Kinesis. Well in short it touted as a very scalable real time data-stream thingy that sings dances and basically makes you line much better. Myself I do not havea use for it but it is part of the system and there is a bug so in I go.

I first had to set things up on the AWS server side with some permission etc the usualal srtuff I also had to run a number of command top build up my Kineses system to a point where I can actually use the 'SubscribeToShard'

I too by now my standard path I created a number of small scripts one each command I was using. Will I eventually had to make my way through 10 actions to get set up. Nothing lost though as for each action I played with I created a test case with 'FullTestMakerLWPCaller.pm'. A good thing as no function tests exists that I can find in PAWS.

Anyway the fist thing I noticed (thanks to Jess https://github.com/castaway who pointed it out) was the boto for the eventual output for this action used something I had not seen before. "eventstream":true


"SubscribeToShardEventStream":{
      "type":"structure",
      "required":["SubscribeToShardEvent"],
      "members":{
        "SubscribeToShardEvent":{"shape":"SubscribeToShardEvent"},
        "ResourceNotFoundException":{"shape":"ResourceNotFoundException"},
        "ResourceInUseException":{"shape":"ResourceInUseException"},
        "KMSDisabledException":{"shape":"KMSDisabledException"},
        "KMSInvalidStateException":{"shape":"KMSInvalidStateException"},
        "KMSAccessDeniedException":{"shape":"KMSAccessDeniedException"},
        "KMSNotFoundException":{"shape":"KMSNotFoundException"},
        "KMSOptInRequired":{"shape":"KMSOptInRequired"},
        "KMSThrottlingException":{"shape":"KMSThrottlingException"},
        "InternalFailureException":{"shape":"InternalFailureException"}
      },
      "eventstream":true
    },

Well if I have learned one thing in my PAWS play- dates it is there are very few one-offs in botot. So I had a snooped about and found it in two others S3 which is mentioned in the bug but also in Pinpoint (one that I actually use).

The second thing I have learned it it is far better to work with boto thatn against it. I have seen similar things in the past so the first thing I need to do is look at the 'Paws::Kinesis::SubscribeToShardEventStream' package and see what I have.


package Paws::Kinesis::SubscribeToShardEventStream;
  use Moose;
  has InternalFailureException => (is => 'ro', isa => 'Paws::Kinesis::InternalFailureException'  );
  has KMSAccessDeniedException => (is => 'ro', isa => 'Paws::Kinesis::KMSAccessDeniedException'  );
  has KMSDisabledException => (is => 'ro', isa => 'Paws::Kinesis::KMSDisabledException'  );
  has KMSInvalidStateException => (is => 'ro', isa => 'Paws::Kinesis::KMSInvalidStateException'  );
  has KMSNotFoundException => (is => 'ro', isa => 'Paws::Kinesis::KMSNotFoundException'  );
  has KMSOptInRequired => (is => 'ro', isa => 'Paws::Kinesis::KMSOptInRequired'  );
  has KMSThrottlingException => (is => 'ro', isa => 'Paws::Kinesis::KMSThrottlingException'  );
  has ResourceInUseException => (is => 'ro', isa => 'Paws::Kinesis::ResourceInUseException'  );
  has ResourceNotFoundException => (is => 'ro', isa => 'Paws::Kinesis::ResourceNotFoundException'  );
  has SubscribeToShardEvent => (is => 'ro', isa => 'Paws::Kinesis::SubscribeToShardEvent'  , required => 1);

1;

well nothing to tell me this is an event stream. If you have been folloing along the next thing I will have to do is play with the templates to add in the value I want. I really only need to tell that this call is an event streram and I have done that many times before with the good old Moose attribute 'class_has'

Seeing that this is an end item my best place to add in my code is in the 'default/object.tt' and this is what I did


…
[% END -%]
++[%- IF (shape.eventstream) %]
++  class_has _event_stream => (is => 'ro', default => 1);
++[%- END %]
1;
[% iclass=shape; INCLUDE 'innerclass_documentation.tt' %]

The neat thing this should also work for the other two APIs but we will see later. Frist a quick recompile;

carton exec builder-bin/gen_classes.pl --classes botocore/botocore/data/kinesis/2013-12-02/service-2.json

and now my class has this;


 has SubscribeToShardEvent => (is => 'ro', isa => 'Paws::Kinesis::SubscribeToShardEvent', required => 1);

  class_has _event_stream => (is => 'ro', default => 1);

1;

So now to use it?

Kinesis uses the 'JsonCaller' and 'JsonResponse' modules so I will be working on those ones but in this post I am really only inserted in 'Response'.

So after a quick poke about in 'lib/Paws/Net/JsonResponse.pm', fortunately I am fairly familiar with this module, I added in this warning near the end of the 'response_to_object' sub;


sub response_to_object {
…
    return Paws::API::Response->new(_request_id => $request_id) if (not $returns);
++  warn("Does this do an evnet stream =".$ret_class->does('_event_stream'));
    my $unserialized_struct = $self->unserialize_response( $response );
…
}

Now just to run my script and lets see if I am on the right track?

Eventually, after about 5 minutes my script returned a response, this was expected though as the API doc says it keeps the Event Stream open only for that long. The response did return a '200' but what I got back was gobbledygook the even the boys over at CuriousMarc would have a hard time decrypting.

It looked soimething like this

'_content' => 'r`▒▒ :message-typeevent{}▒▒k}▒ejI▒z-json-1.1 :message-typeevent{"ContinuationSequenceNumber":"49604106570538379893614004884743816156465460913031348226","MillisBehindLatest":0,"Records":[]}-▒▒▒ejI▒ :event-typeSub:message-typeevent...

…

but I did see a successful line of debuggin as well

Does this do an evnet stream =1

So a few good things

I am getting succes back from AWS

There is some JSON in the stuff I amd getting back from AWS

my little patch will let me handle the wacked out content

I did take a very close look at was returned and I noticed that the 'content-type' was something called 'application/vnd.amazon.eventstream' So It looks like I have some research to do. Really going to have to put my thinking Pom on for this one