Paws for Fun and Profit. Part the Second
So today I am going to start some actual work on Paws by playing with Paws::S3. For those that do not know S3 is the Amzazon's Simple Storage System a place where we AWS types can store just about anything we want up on the cloud.
One part of S3 is the ability to change the storage state/class of an artifact. It all depends on what you are doing with the artifact. If the artifact needs to be shared across the web, like an image, you would just put it in the 'Standard' class. If you are archiving artifacts for safe keeping and there is no requirement to get them back right away you could put them int the 'Glacier' class which is much cheaper.
Today I have a need to store an image artifacts in 'Standard' class on S3 so they can be incorporated into on line reports. Simple enough but it is also expensive in the long run to leave the images in that class as some 6~7k images are uploaded daily.
Fortunately very few of these images are accessed more than 90 days after their creation so any ones older that that I can move into 'Glacier' and save some money.
For statutory reasons the images have to be kept for a minimum of 5 years and the rub comes when a customer wants a document that is in 'Glacier'. At present someone had to do this by hand with the AWS online console, not a long term solution, we need a way for the customer to do this.
Now what does this have to do with Paws?
Well there is a command in Paws called 'RestoreObject' so the plan is to write a web page or two where the customer can request an image from the archive and use Paws to restore the object.
Well first I made a pull request from the master to get a local copy of repo I happen to have on tag already on my branch so I will use that one.
git clone https://github.com/byterock/aws-sdk-perl.git –branch=s3ObjectTagging
after that is all done all I need to do is
carton install
which will run for a little while as it installs all the needed packages.
When done I do a quick
$ make pull-other-sdks
which pull is boto and botocore (the json data file used to create the classes)
Once that is done I am ready to go.
Now I am really just interested in the 'S3' part of AWS right now so I will recompile just that part like this
carton exec builder-bin/gen_classes.pl --classes botocore/botocore/data/s3/2006-03-01/service-2.json
any files that are generated from this are found in the 'auto-lib' dir. In my case I can see that most of the files in the 'auto-lib/Paws/S3/' have been rebuild
Now for a quick test script;
#!/usr/bin/env perl
use Data::Dumper;
use strict;
use Paws;
my $s3 = Paws->service('S3', region => 'us-east-1');
my $restore_output = $s3->RestoreObject(
Bucket => 'abucket.someplace',
Key => 'upload/image_1033233_275731_69069.jpg',
);
warn("out put=".Dumper($restore_output));
In the above case I used just the bare minimum of non optional paras just to see it if will work. The are Bucket and Key, which represent where it artifact is stored (bucket) and what it is named (key).
On my first run I got;
Bad Request
Ok got some Moose poop. There is not much on the info side of things and for now I am not going to worry about the;
Paws::S3 is not stable / supported / entirely developed at /ws-sdk-perl/auto-lib/Paws/S3.pm line 2.
but the the next line;
'Bad Request'
That is telling me something that I was sending AWS does not like.
Fortunately, I have been playing with Paws for a while and I know how to read the Moose-poop and figure out what to do.
First;
Paws::Net::RestXMLResponse::process('Paws::Net::RestXMLResponse=HASH(0x3c811b0)', 'Paws::S3::RestoreObject=HASH(0x3cacd58)', 'Paws::Net::APIResponse=HASH(0x3db19b0)') called at /aws-sdk-perl/lib/Paws/Net/Caller.pm line 46
this line tells me that whatever response was coming back from AWS in being handled in the ' process' sub in 'Paws::Net::RestXMLResponse' so lets add in a little debug in this file '/aws-sdk-perl/lib/Paws/Net/Caller.pm '
sub process {
my ($self, $call_object, $response) = @_;
use Data::Dumper;
warn("process response=".Dumper($response));
if ( $response->status >= 300 ) {
The nice thing about Paws is I do not need to recompile the code when I make a change to this class as this is not one of the ones that is auto-generated. So after this change I get this;
Paws::S3 is not stable / supported / entirely developed at /home/scolesj/aws-sdk-perl/auto-lib/Paws/S3.pm line 2.
process response=$VAR1 = bless( {
'content' => '<?xml version="1.0" encoding="UTF-8"?>
<Error>
<Code>
MissingRequestBodyError
</Code>
<Message>
Request Body is empty
</Message>
<RequestId>
B3F753E31141E1F
</RequestId>
<HostId>m5AqUewbyuJ7uUFRFNqnlGIytEUt5sYzBZiIH3vQWxYByNE=
</HostId>
</Error>',
'headers' => {
'x-amz-request-id' => 'B3F753E31141E1F0',
'date' => 'Fri, 20 Sep 2019 20:32:56 GMT',
'server' => 'AmazonS3',
'x-amz-id-2' => 'mJENXVMOzoRApmUb0G4tiCcGDVobP5AqUewbyuJ7uUFRFNqnlGIytEUt5sYzBZiIH3vQWxYByNE=',
'content-type' => 'application/xml',
'transfer-encoding' => 'chunked',
'connection' => 'close'
},
'status' => '400'
}, 'Paws::Net::APIResponse' );
Bad Request
...
So that at least tell me a little more as the 'Code' returned is 'MissingRequestBodyError' with the 'Message' 'Request Body is empty'.
To move this along I will have to see what I am sending as a request. I could put a 'caller' function in my debug code to see where this is being called but I do not have to.
Paws code is very well though out and each 'Response' class has it own 'Caller' class. So I can take a good guess and look and see if there is a 'Paws::Net::RestXmlCaller' class. Sure enough there is in 'lib/Paws/Net/RestXmlCaller.pm'
A look in that file and I found a that is a Moose Role.
package Paws::Net::RestXmlCaller;
use Paws;
use Moose::Role;
…
package Paws::S3;
warn "Paws::S3 is not stable / supported / entirely developed" unless $ENV{'PAWS_SILENCE_UNSTABLE_WARNINGS'};
…
with 'Paws::API::Caller', 'Paws::API::EndpointResolver', 'Paws::Net::S3Signature', 'Paws::Net::RestXmlCaller';
…
as a bonus I also found where that other waring is coming from. So another guess at what file to look into and I have a look at the 'Paws::API::Caller'. I did not find what I was looking for in that file as it looks one level higher that I need so I went back to ' RestXmlCaller' and found a likely place in the 'prepare_request_for_call' sub.
sub prepare_request_for_call {
my ($self, $call) = @_;
my $request;
…
use Data::Dumper;
warn("prepare_request_for_call request=".Dumper($request));
return $request;
}
lets see what happens
Paws::S3 is not stable / supported / entirely developed at /home/scolesj/aws-sdk-perl/auto-lib/Paws/S3.pm line 2.
prepare_request_for_call request=$VAR1 = bless( {
'uri' => '/dev.cargotel.storage/lhf/insp/attachments/assigned/condition_1_1033233_275731_69069.jpg?restore',
'headers' => bless( {
'x-amz-content-sha256' => 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855',
'::std_case' => {
'x-amz-content-sha256' => 'X-Amz-Content-Sha256',
'x-amz-date' => 'X-Amz-Date'
},
'host' => 's3.amazonaws.com',
'content-md5' => '1B2M2Y8AsgTpgAmY7PhCfg==',
'authorization' => 'AWS4-HMAC-SHA256 Credential=AKIAJ25UFIFHZ4WML4BQ/20190920/us-east-1/s3/aws4_request,SignedHeaders=content-md5;date;host;x-amz-content-sha256;x-amz-date,Signature=2d032abff73f1b79594387d05c5514590bcd644a85e87bda06b96c98f2cc9a27',
'x-amz-date' => '20190920T205507Z',
'date' => '20190920T205507Z'
}, 'HTTP::Headers' ),
'method' => 'POST',
'url' => 'https://s3.amazonaws.com/dev.cargotel.storage/lhf/insp/attachments/assigned/condition_1_1033233_275731_69069.jpg?restore',
'parameters' => {
'Bucket' => 'dev.cargotel.storage',
'Key' => 'lhf/insp/attachments/assigned/condition_1_1033233_275731_69069.jpg'
},
'content' => '',
'date' => '20190920T205507Z'
}, 'Paws::Net::S3APIRequest' );
process response=$VAR1 = bless( {
...
Ok I see nothing on the 'content' part so that is why I am getting the error so I think I need to add a little more into my basic call.
Looking the Pod again I noticed that the 'RestoreRequest' dose not have an optional at the end (way at the bottom) so I will add that in. Though I am surprised it did not thow a 'validation error';
Bucket => 'abucket.someplace',
Key => 'upload/image_1033233_275731_69069.jpg',
++RestoreRequest => {}
);
'content' => '<RestoreRequest><RestoreRequest>',
in the content but the same 400 error but this time my errors is
<Error>
<Code>MalformedXML</Code>
<Messagg>
The XML you provided was not well-formed or did not validate against our published schema
</Message>
<RequestId>
CBAAF3631DD78E4A
</RequestId>
<HostId>
Fd6fDjtUv9+JMnJlqoSjrfgqZYfIQwNswzGJACj8ue1zNAp0CX/yfX2Vie+McHxWs=
</HostId>
</Error>
So still missing something and to find out what I have to look at the AAWS PI documentation, POST Object Restore. There I find that Days is 'Required' if you are restoring an archive. Another quick change
Bucket => 'abucket.someplace',
Key => 'upload/image_1033233_275731_69069.jpg',
--RestoreRequest => {}
++RestoreRequest => {Days=>2}
);
and then I give it a quick run and get
process response=$VAR1 = bless( {
'content' => '',
'headers' => {
'server' => 'AmazonS3',
'date' => 'Fri, 20 Sep 2019 23:18:30 GMT',
'content-length' => '0',
'x-amz-version-id' => 'null',
'x-amz-id-2' => 'F/Aqv8iKKyDxL9zJdqkDtm4iBgKtL2B+AXlEiNE7HzjwcDaLroGM7duOZTCCIYT/ITxFqlu3teU=',
'x-amz-request-id' => '9C4494F17CFA569D'
},
'status' => '202'
}, 'Paws::Net::APIResponse' );
Success!
Well that was easy was just a case of me not having the correct data in place and the POD being a little lax on what was a required field or not.
I really want to contributed something for today's post and I think I will look into updating the POD but that is another story.
Long time Perl guy, a few CPAN mods allot of work on DBD::Oracle and a few YAPC presentations
Leave a comment