Well I have said it in the past I!=SysAdmin and with the problems I have had of late I think I might as well share with other so they will not have the same three very bad days I just had.

To make a short story a little longer it started when the development box I usually work-with was rebuild after a fatal crash on the rather tired box it was on.

Things where fine on the new box the old code worked and we seemed to see no problems a few runs of cpanm would not fix. The problems started after we got caught up with the backed up day to day production fixes and I started to play with some new things that I am using.

To be specific I am making more and more use of Amazon.AWS services these days (yes with perl) and the two problem children where SNS and SQS. Now the code in question was using Amazon::SQS::Simple and Amazon::SNS but these where buried way back in the code as a small part of a much larger system, that uses other new goodies like 'MongoDB' and 'RabbitMQ'

So I go and try my new code and get an error like this

ERROR [try ]: On calling CreateQueue: 500 Can't connect to (certificate verify failed) at /usr/lib/perl5/site_perl/5.8.8/Replay/EventSystem/ line 169.

Now this new system is highly dependent on threading and we has issues in the past so I went though the usual checks and rechecks of all the installed software and synched up the Perl mods on the new box to be the same as the old box.

As an aside I had to do a number of removal and reinstall of Mons so thank goodness for the CPANM -U saved me a so much time.

Well still having the same problems so I went to the '/etc/pki/tls' dir and checked to make sure everything was the same between the two boxes and it was so I dove deeper into the code.

I spent a good deal of time ensuring threading was working correctly and that is was not some other config issue and in the send by tracing back I found the offending and it was doing nothing more than this

   my $sqs = Amazon::SQS::Simple->new(

So using that block of code I tried it in a little test program and ploush the same error

So I took it back once more and looked in SQS::Simple to see what it was doing and I recreated it with this block of code

       my $uri = URI->new($service);
        $uri->query(join('&', map { $_ . '=' . URI::Escape::uri_escape_utf8($args->{$_}, '^A-Za-z0-9\-_.~') } sort keys %$args ));
        # create signature
        $args->{'Signature'} = hmac_sha256_base64(join("\n", 'POST',
          $uri->host, $uri->path, $uri->query), "dfskljdfaskljdfskljdfskljdfskljrewdfskldfas");
        # padding
        while (length($args->{'Signature'}) % 4) {
                $args->{'Signature'} .= '=';
        # rewrite query string
        $uri->query(join('&', map { $_ . '=' . URI::Escape::uri_escape_utf8($args->{$_}, '^A-Za-z0-9\-_.~') } sort keys %$args ));
        my $response = LWP::UserAgent->new->post($service, 'Content' => $uri->query);

in my test script and ploush it died as well.

So I tried the direct approach with just 'LWP::UserAgent'
 $response = LWP::UserAgent->new->post(


'Content' =>"AWSAccessKeyId=dfsdfskj3kjdfskldfspomkldfsklj&Action=CreateTopic&Name=sandbox_jps_replay_origin&Signature=%2Bbl0N91Ju3JKpF25Zuwx7cgk7Rj20Lmz37tQrgKhx5s%3D&SignatureMethod=HmacSHA256&SignatureVersion=2&Timestamp=2015-03-25T21%3A05%3A24.000Z"

and it died too.

I even tried the direct approach with curl

curl --request POST --data "AWSAccessKeyId=dfskljdfskljdfskljdfskljdfs&Action=CreateTopic&Name=sandbox_jps_replay_origin&Signature=dfskljdfskljdfskljdfskljdfskljdfskljdfsSignatureMethod=HmacSHA256&SignatureVersion=2&Timestamp=2015-03-25T20%3A48%3A53.000Z"
using the values generated in my test script and bingo the curl worked so I could get into AWS and to add even more frustration both Amazon::SNS and Amazon::SQS::Simple tested 100% ok this was really getting to me

Well after a very long time testing many different things we (I brought in another heavy hitter for this one, thanks again skylos) we found that at one point in the chain 'Mozilla::CA' was being hit and that is where the CA certificates where being hit.

a quick
'perl -MMozilla::CA -e 'print $Mozilla::CA::VERSION'
on the prod server and then on the Dev server
'perl -MMozilla::CA -e 'print $Mozilla::CA::VERSION'
ah so a quick
cpanm Mozilla::CA@20130114


it was fixed.

So if you ever get the above this little post will save your day.


So, backrevving Mozilla::CA fixed it. I'd think you would want to find out why the newer version broke, and repair that problem.

Its a puzzle, right? On further research, the named authority for the certificate at the base of the chain is thus:

Root 3
VeriSign Class 3 Primary CA - G5
Description: This root CA is the root used for Symantec Extended Validation Certificates and should be included in root stores. During Q4 2010 this root will also be the primary root used for all Symantec SSL and Code Signing certificates.

Country = US
Organization = VeriSign, Inc.
Organizational Unit = VeriSign Trust Network
Organizational Unit = (c) 2006 VeriSign, Inc. - For authorized use only
Common Name = VeriSign Class 3 Public Primary Certification Authority - G5
Serial Number: 18 da d1 9e 26 7d e8 bb 4a 21 58 cd cc 6b 3b 4a
Operational Period: Tue, November 07, 2006 to Wed, July 16, 2036
Certificate SHA1 Fingerprint: 4e b6 d5 78 49 9b 1c cf 5f 58 1e ad 56 be 3d 9b 67 44 a5 e5
Key Size: RSA(2048Bits)
Signature Algorithm: sha1RSA

And the appropriate pem block is present in both cert.pem clumps (MIIE0zCCA7ugAwIB...ao7WNq), the date-time on the systems is within tolerance. Mozilla::CA consists of entirely 'load the cert.pem file' which can't really have a bug and work for anything. My hypothesis are exhausted.

-heavy hitter

I think I finally chewed the problem to a solution.

Leave a comment

About byterock

user-pic Long time Perl guy, a few CPAN mods allot of work on DBD::Oracle and a few YAPC presentations