Lock me away

I am in the process of writing Yet Another Job Queue, Cantella::JobQueue.

I decided from the beginning that I wanted a well-defined API for the storage layer, so that as different back ends would be easy to write and swap around. While writing the tests for the first one, I encountered a problem that’s been haunting me since: how to deal with locks.

Originally, I imagined a job would either be locked, or not locked and manager processes would be in charge of releasing locks if workers died / failed. Then I started thinking, what if a manager process dies? Who releases the locks? So I decided to add time-based automatic lock expiration. This introduced a new problem, a possible race condition. What if a job takes so long that it’s lock times out before the job is done? What do I do then?

A possibility is to have a manager process / thread that monitors its workers and updates the lock-expiry time, which is the route I am likely to take. I don’t particularly like it, but I can’t come up with anything better right now.

I’d love to hear from anyone that has any experience with this kind of problem.


TheSchwartz's behavior is to provide a default expected finish time, and if a the completed call hasn't occurred by then, to free the job. I believe that Beanstalkd works the same way. I'm not quite sure what architecture you're imagining -- here's your vanished post from yesterday, and please contact me, Aaron, or Dave Cross about merging your accounts etc. -- but if your manager dies, isn't it best to assume that the queue is defunct? I was assuming that Cantella would just be a separate process, a la Starling, Beanstalkd, Memcacheq, etc., listening on a port somewhere.

We moved our job queue for Socialtext over from some shitty home-grown system to TheSchwartz and it's super awesome for us. We use TheSchwartz::Moosified which is actively maintained.

It seems really great. Sorry to hear you need to re-invent another system.

Leave a comment

About Guillermo Roditi

user-pic I blog about Perl.