byterock

Well I have said it in the past I!=SysAdmin and with the problems I have had of late I think I might as well share with other so they will not have the same three very bad days I just had.

To make a short story a little longer it started when the development box I usually work-with was rebuild after a fatal crash on the rather tired box it was on.

Things where fine on the new box the old code worked and we seemed to see no problems a few runs of cpanm would not fix. The problems started after we got caught up with the backed up day to day production fixes and I started to play with some new things that I am using.

To be specific I am making more and more use of Amazon.AWS services these days (yes with perl) and the two problem children where SNS and SQS. Now the code in question was using Amazon::SQS::Simple and Amazon::SNS but these where buried way back in the code as a small part of a much larger system, that uses other new goodies like 'MongoDB' and 'RabbitMQ'

So I go and try my new code and get an error like this

ERROR [try ]: On calling CreateQueue: 500 Can't connect to sqs.us-east-1.amazonaws.com:443 (certificate verify failed) at /usr/lib/perl5/site_perl/5.8.8/Replay/EventSystem/AWSQueue.pm line 169.

Now this new system is highly dependent on threading and we has issues in the past so I went though the usual checks and rechecks of all the installed software and synched up the Perl mods on the new box to be the same as the old box.

As an aside I had to do a number of removal and reinstall of Mons so thank goodness for the CPANM -U saved me a so much time.

Well still having the same problems so I went to the '/etc/pki/tls' dir and checked to make sure everything was the same between the two boxes and it was so I dove deeper into the code.

I spent a good deal of time ensuring threading was working correctly and that is was not some other config issue and in the send by tracing back I found the offending and it was doing nothing more than this

 
   my $sqs = Amazon::SQS::Simple->new(
        'dfshjdasjw',
        "ldfskljdfaskljKLFSKLHafkljdfaskljdfaskljdfasklj",
        {Endpoint=>"https://sns.us-east-1.amazonaws.com",}
    );

So using that block of code I tried it in a little test program and ploush the same error

So I took it back once more and looked in SQS::Simple to see what it was doing and I recreated it with this block of code

 
       my $uri = URI->new($service);
        $uri->path('/');
        $uri->query(join('&', map { $_ . '=' . URI::Escape::uri_escape_utf8($args->{$_}, '^A-Za-z0-9\-_.~') } sort keys %$args ));
        # create signature
        $args->{'Signature'} = hmac_sha256_base64(join("\n", 'POST',
          $uri->host, $uri->path, $uri->query), "dfskljdfaskljdfskljdfskljdfskljrewdfskldfas");
        # padding
        while (length($args->{'Signature'}) % 4) {
                $args->{'Signature'} .= '=';
        }
        # rewrite query string
        $uri->query(join('&', map { $_ . '=' . URI::Escape::uri_escape_utf8($args->{$_}, '^A-Za-z0-9\-_.~') } sort keys %$args ));
        my $response = LWP::UserAgent->new->post($service, 'Content' => $uri->query);

in my test script and ploush it died as well.

So I tried the direct approach with just 'LWP::UserAgent'

 

 $response = LWP::UserAgent->new->post(


"https://sns.us-east-1.amazonaws.com",

'Content' =>"AWSAccessKeyId=dfsdfskj3kjdfskldfspomkldfsklj&Action=CreateTopic&Name=sandbox_jps_replay_origin&Signature=%2Bbl0N91Ju3JKpF25Zuwx7cgk7Rj20Lmz37tQrgKhx5s%3D&SignatureMethod=HmacSHA256&SignatureVersion=2&Timestamp=2015-03-25T21%3A05%3A24.000Z"

and it died too.

I even tried the direct approach with curl

curl --request POST --data "AWSAccessKeyId=dfskljdfskljdfskljdfskljdfs&Action=CreateTopic&Name=sandbox_jps_replay_origin&Signature=dfskljdfskljdfskljdfskljdfskljdfskljdfsSignatureMethod=HmacSHA256&SignatureVersion=2&Timestamp=2015-03-25T20%3A48%3A53.000Z" https://sns.us-east-1.amazonaws.com
#1427316636

using the values generated in my test script and bingo the curl worked so I could get into AWS and to add even more frustration both Amazon::SNS and Amazon::SQS::Simple tested 100% ok this was really getting to me

Well after a very long time testing many different things we (I brought in another heavy hitter for this one, thanks again skylos) we found that at one point in the chain 'Mozilla::CA' was being hit and that is where the CA certificates where being hit.

a quick

'perl -MMozilla::CA -e 'print $Mozilla::CA::VERSION'

20130114

on the prod server and then on the Dev server

'perl -MMozilla::CA -e 'print $Mozilla::CA::VERSION'

20141217

ah so a quick

 

cpanm Mozilla::CA@20130114

and

it was fixed.

So if you ever get the above this little post will save your day.