AWS Lambdas, Furl & LWP
At re:Invent 2018 AWS announced custom Lambda runtimes. This makes it possible to create Perl based Lambdas. Although theoretically it was possible to create a Perl Lambda prior to this announcement by invoking a shell from Python for example, the new custom runtimes make it possible to use almost any language to create Lambdas.
I've been developing a Perl based custom runtime framework so that creating a Perl Lambda is pretty much as simple as:
package MyLambda;
use parent qw/Amazon::Lambda::Runtime/;
sub handler {
my $self = shift;
my ($event, $context) = @_;
$self->get_logger->info("...Perl has just joined the party!");
return "Hello World";
}
Implementing a custom runtime for Perl seems rather straight forward, however there a lot of mysteries regarding the way Lambdas are actually invoked and executed. According to the AWS documentation, one should create an event loop waiting for Lambda events by calling an API endpoint. In experimenting with various tools I noticed that using Furl vs LWP produced an interesting reveal...it appears AWS can freeze time! Huh?
Well, not really, but certainly that's one explanation. Amazon seems to be able to do pretty much anything they want to do. Actually though, I think the phenomenon I observed has more to do with the way timeouts are implemented in Furl vs LWP - anyone who can verify or refute my theory is encouraged to chime in and educate me on this topic.
When using Furl it seems that timeouts are implemented based on the clock on the wall, however for LWP it appears that timeouts are based on elapsed processing time of the function. I've arrived at this conclusion by noting the behavior of Lambdas that are called shortly after one another. AWS maintains a Lambda environment (most likely using some kind of container - actually it's called Firecracker) for your Lambda for some period of time after invocation in order to minimize startup time and presumably to make Lambdas more efficient. One can capitalize on that behavior by caching certain data that might otherwise be expensive to retrieve each time. While you can't guarantee the data will be present when your Lambda is invoked, you can take advantage of it if it is present. Many people do this in their Python and Node Lambdas.
So let's say the timeout for Furl is set to 30 seconds. When I make a call to my Lambda runtime which uses Furl to fetch the event...all good..event retrieved. Now we continue in our event loop and make another call to the endpoint waiting for an HTTP response. We would expect that if no event were present the Lambda would timeout - however here's where it gets weird. The Lambda actually stops executing after you return a response and make a request to the endpoint used for getting events. The INVOCATION_NEXT request to fetch the next event appears to trigger the end of the billing cycle for that Lambda and puts your Lambda to sleep or at least in suspended animation.
There appear to be two "stopped" conditions for Lambdas. In condition A, right after a successful call, the Lambda appears to go to sleep but still maintains its environment. In condition B the Lambda environment is torn down and another call to the Lambda will recreate the runtime environment.
Now back to Furl vs LWP. When Furl is used and the Lambda is "woken", Furl times out and returns a 500 (internal response). When LWP is used and the Lambda is "woken", LWP returns the new event and does not present a timeout. Hmmm...curious. I'm therefore left to conclude that LWP is using relative execution time and Furl is using wall clock time. Or there is some other explanation or implementation detail (alerts?) that I have not thought of.
I'll be posting my project on github soon if anyone is interested. So far, I'm confident that Perl Lambdas are going to be viable, fun and perhaps a way to see more Perl in the AWS environment!
I blog about Perl and Bedrock.