Perl Zombie Ate My Brain
Well something like that. The whole story is that I wanted to add in some new code on a rather old web-page. As the new code took some time to run I wanted to do it as an asynchronous call. So my new code would run off line and not slow down the already slow response time.
So I being a module sort of guy I had a try with Async and all seemed to work correctly in dev so we moved it into production and it seemed to work fine
Until
our Nagios started to complain and a quick 'top' gave us
Tasks: 1515 total, 1234 running, 670 sleeping, 6 stopped, 654 zombie
Well a after a pull-back of the code and then a clean up with a number of kill commands I got rid of them.
So what was the problem??
well the code was simple enough
sub emit_async {
my $self = shift;
my ($emitter) = @_;
use Async;
my $proc = Async->new(sub {
$self->emit($deligater);}
);
}
What I quickly discovered was that in dev each web page was compiled at call time, as you would expect, unless you like to restart you sever allot, while production was set up to pre-compile and then use a number of workers.
It appears that my little addition was forking the process and adding in the zombies and many more process as I did not have a 'ready' call on my $proc object to get into the kernel tell it I as finished with my process.
It worked in dev because each hit meant the end of the parent process and then all of its children.
So not having can of 'Zombie Off' handy
to spray over my code so I had to come up with a solution.
I did not like a suggested Mod_Perl solution where you add in 'CORE::exit(0);' in some places as I was not sure what the impact might of have been and I was not sure exactly where to add it into my code.
As a Zombie process is just a child with no parent one solution is to ensure you never have any child process.
So all I had to do is change my code like this
sub emit_async {
my $self = shift;
my ($emitter) = @_;
use Async;
my $proc = Async->new(sub {
my $child_proc = AsyncTimeout->new(sub {
$self->emit($deligate_emitter);}
);
}
);
if ($proc->ready) {
# the code has finished executing
if ($proc->error) {
# something went wrong
}
else {
my $result = $proc->result; # The return value of the code
}
}
}
So I added in a grandchild fork to my child fork. What happens is the first child fork executes immediately and exits, which I then clean up the 'ready' call in the parent (the web page). Now that the child no longer exists the grandchild process will just disappear when finished as it now has no parent either.
So very quick 'fire and forget' Perl calls and no zombies on my server
If you aren't ever going to wait for your child processes, why not get rid of the zombies by doing $SIG{CHLD} = 'IGNORE'? ( http://perldoc.perl.org/functions/fork.html , paragraph starting with "If you fork without ever waiting on your children, you will accumulate zombies.")