Thoughts::Perl::Random

OK, the tag “worst Perl” might be overstating the case, but we ran into a funny error message yesterday from a program that runs on all of our production equipment. As each lot is tracked into a piece of equipment, the program running on it contacts our MES (manufacturing execution system) to verify information about the lot. Because the MES used to run on a really slow server, timeouts were common, so at some point the function to look up a lot was wrapped in a function to retry a few times in that event. The MES server has since been upgraded, but we never took out the retry functionality. Here's the code in question:

our $MAX_ATTEMPTS = 5;

sub lookup_lot {
  # Some terrifying code goes here. It throws an
  # exception on all errors including timeouts.
}

sub lookup_lot_with_retry {
  our $attempts = 0;
  do {
    our $result = eval { lookup_lot(@_) };
    if ($@) {
      # Exceptions besides timeouts are re-thrown.
      die $@ if $@ !~ m/:[ ]Timed[ ]out[ ]/;
    }
    else {
      return $result; # no error means success
    }
    # The call to lookup_lot() timed out; try again.
  } while (++$attempts < $MAX_ATTEMPTS);
  die "Failed operation after $attempts attempts\n";
}

All that is fine, because it’s actually the corrected version I deployed. In the previous version, $attempts was accidentally declared in a global scope, right below where $MAX_ATTEMPTS was set. Did I mention the program that contains this code run continuously for months at a time, tracking in lot after lot after lot? All this led to the following log entry:

2017-03-02,16:23:12,ERROR,Failed operation after 49892 attempts

Oops.