The project is growing up like Alice after drinking poison. You expected that it should handle only a few requests and eventually it has to serve many more requests at same moment. Or another scenario – you need to run at once a few tasks and make sure they all are going to be completed and you want system notifies you about their progress. That’s where job queue comes to help you. In specific jobs queue is something we have to use in our symfony-driven RSS reader – we want to make sure parsing of each feed source can be done independently and in time.
It seems there are a few solutions for this problem if you use symfony:
1. Symfony 1.0 plugin sfJobQueue. So you’d have to rewrite it a bit to make it working with newer symfony versions (especially if you don’t plan to use Propel). Plugin has quite basic functionality but advantage of its using – built-in backend modules to manage jobs progress (so good for quick solutions).
2. Zend provides a few ways to solve queuing problem. Zend_Queue is unified class which can use a different messaging platforms like Amazon SQS, Zend Platform Queue, etc. In order to use it with symfony you’d have install Zend bridge plugin or add classes directly into lib folder. Here is available simple sample of its using for mass sending emails with MSSQL
Another approach from Zend is Job Queue for Zend Platform which provides really powerful bunch of queue features, just have a look on this sample:
$job_queue = new ZendAPI_Queue('gollum.zend.office'); $job_queue->login('1234'); $job = $job_queue->getJob(8); $job->setRecurrenceData(3600, time()+3600*24); $job->setJobDependency(16); // This job will perform only after job 16 was successfully executed $job_queue->updateJob($job); // The job queue will update job #8 with the new properties of the given Job object |
Obviously you’d have install Zend Platform which makes this solution less simple than others (I’ve never used it so probably it does not really complicated). For interested I’d also recommend to have a look on this analytic article: Zend Job Queue.
3. Gearman is scalable queue solution or like they describe their-self:
a system to farm out work to other machines, dispatching function calls to machines that are better suited to do work, to do work in parallel, to load balance lots of function calls, or to call functions between languages.
So in some sense this solution is the most perspective as you can be sure that you’ll be able to handle as much requests as you need. You’d have to use this PHP API if you go with gearman.
There are a few notes regarding it though. So firstly when you install gearman daemon you’ll most probably will have to get libevent lib, here are the installation steps:
wget http://launchpad.net/gearmand/trunk/0.12/+download/gearmand-0.12.tar.gz
tar zxf gearmand-0.12.tar.gz
yum install libevent-devel
./configure
make
make install
Also current php extension 0.6.0 can’t be compiled properly (it seems to be working fine only under PHP 5.3) so you’d have to go either with PEAR package (which is still alpha though):
pear install Net_Gearman-0.2.3
or install previous version of php extension to compile it properly on servers with php < 5.3. On everything else we refer you to gearman PHP API docs in order to start using it. 4. And eventually if you think that none of provided solutions is suitable you can always write own one using this article which provides general ideas on job queue implementation: http://stut.net/2009/05/29/php-job-queue/
So there is quite big amount of approaches to get queuing problem solved. We hope you’ll drop us comment if you know any other approach. Personally I would vote for gearman as this looks like pretty promising solution so developing infrastructure around it would be good investment.
Finally here are some related subjects which are kind of out of scope of this article but still can be interesting for you:
Asynchronous work in backend:
http://groups.google.com/group/symfony-users/browse_thread/thread/afdda6e5f92d8f21?pli=1
symfony queue requests sent by a single browser session:
http://forum.symfony-project.org/index.php/m/77832/