Lock me away
I am in the process of writing Yet Another Job Queue, Cantella::JobQueue.
I decided from the beginning that I wanted a well-defined API for the storage layer, so that as different back ends would be easy to write and swap around. While writing the tests for the first one, I encountered a problem that’s been haunting me since: how to deal with locks.
Originally, I imagined a job would either be locked, or not locked and manager processes would be in charge of releasing locks if workers died / failed. Then I started thinking, what if a manager process dies? Who releases the locks? So I decided to add time-based automatic lock expiration. This introduced a new problem, a possible race condition. What if a job takes so long that it’s lock times out before the job is done? What do I do then?
A possibility is to have a manager process / thread that monitors its workers and updates the lock-expiry time, which is the route I am likely to take. I don’t particularly like it, but I can’t come up with anything better right now.
I’d love to hear from anyone that has any experience with this kind of problem.
I blog about Perl.
TheSchwartz's behavior is to provide a default expected finish time, and if a the completed call hasn't occurred by then, to free the job. I believe that Beanstalkd works the same way. I'm not quite sure what architecture you're imagining -- here's your vanished post from yesterday, and please contact me, Aaron, or Dave Cross about merging your accounts etc. -- but if your manager dies, isn't it best to assume that the queue is defunct? I was assuming that Cantella would just be a separate process, a la Starling, Beanstalkd, Memcacheq, etc., listening on a port somewhere.
Aha! Thanks for the tip. I am now looking at Beanstalk, which I did not know about. Thanks a lot!
We moved our job queue for Socialtext over from some shitty home-grown system to TheSchwartz and it's super awesome for us. We use TheSchwartz::Moosified which is actively maintained.
It seems really great. Sorry to hear you need to re-invent another system.