Web Queues

When writing a web application, developers usually think in the “user request” -> “server processing” -> “response” cycle. And they are right to think so, since this is how HTTP works. However sometimes you need to accomplish tasks outside the scope of this cycle, to run scheduled jobs, long running jobs or callback jobs.

There are many ways to go about this, but the best way IMO, is to have a separate worker listening to a queue. What you do is that you create a queue, and whenever you want to run a job outside the scope of the web cycle, you push a message to the queue. Separate workers would exist and listen to that queue, and whenever there is a new job, they will fetch that job and run it.

So far, I have tried using three different systems for queuing jobs: ActiveMQ, Amazon SQS, and most recently Beanstalk. I have to say that my personal preference is Beanstalk. ActiveMQ was a complete pain to install and configure, and I actually gave up on it before getting it to run.

Amazon SQS has no setup or configuration since it is a service offered by Amazon. However, not having the queue locally means slower response times, and you don’t get that feeling of having full control over your jobs. More importantly, for some reason, some jobs would get completely lost and I can’t figure where they went. Having to manually reinsert jobs into queues every so often became tedious quickly, and after a year of this we started looking for an alternative.

Beanstalk is an extremely fast and lightweight queuing server, has almost no setup overhead (you have to compile from source but that’s easily done on mac or linux), and can take on some serious load. The only problem I found with beanstalk is that it desperately lacks good documentation online, as I find myself resorting to the source code to figure out what functionality it offers.

As such, I will write articles about Beanstalk whenever I find a new feature that I didn’t know about to help spread the knowledge about this amazing queuing server. Stay tuned.

Advertisement