Wez Furlong

Browse archives
Conference Presentations
Subscribe. (circulation 746)
Comments. (circulation 2)

Search powered by Google

I am Wez Furlong, Director of Engineering at Message Systems. My team is responsible for the fastest MTA on Earth.

I'm also a PHP Core developer and OpenSource contributor, residing in Maryland with Juliette, Xander and Lily. (read more)

26th October 2006 @ 17:19 EDT

One of the other things I've been looking it as ways to implement background processing in PHP. In my recent talk on sending mail from php I mention that you want to avoid sending mail directly from a web page. A couple of people have asked me how to implement that, and one of the suggestions I have is to queue your mail in a database table and have some other process act on that table.

The idea is that you have a PHP CLI script that, in an infinite loop, sleeps for a short time then polls the database to see if it needs to do some work. While that will work just fine, wouldn't it be great if the database woke you up only when you needed to do some work?

I've been working on a patch originally contributed by David Begley that adds support for LISTEN/NOTIFY processing to the Postgres PDO driver. With the patch you can write a CLI script that looks a bit like this:

<?php
   $db = new PDO('pgsql:');
   $db->exec('LISTEN work');
   dispatch_work();
   while (true) {
      if (is_array($db->pgsqlGetNotify(PDO::FETCH_NUM, 360))) {
          dispatch_work();
      }
   }
?>

This script will effectively sleep for 360 seconds, or until someone else issues a 'NOTIFY work' query against the database, like this:

<?php
   $db->beginTransaction();
   $q = $db->prepare('insert into work(...) values (...)');
   $q->execute($params);
   $db->exec('NOTIFY work');
   $db->commit();
?>

When the transaction commits, the CLI script will wake up and return an array containing 'work' and a process id; the script will then call dispatch_work() which is some function that queries the database to find out exactly what it needs to do, and then does it.

This technique allows you to save CPU resources on the database server by avoiding repeated queries against the server. The classic polling overhead trade-off is to increase the time interval between polls at the cost of increased latency. The LISTEN/NOTIFY approach is vastly superior; you do zero work until the database wakes you up to do it--and it wakes you up almost immediately after the NOTIFY statement is committed. The transactional tie-in is nice; if something causes your insert to be rolled back, your NOTIFY will roll-back too.

Once PHP 5.2.0 is out the door (it's too late to sneak it into the release candidate), you can expect to see a PECL release of PDO::PGSQL with this feature.

by Wez Furlong in .
Post a comment
26th October 2006 @ 22:56 EDT

Does MySQL also support this?

by Jacques in .
27th October 2006 @ 02:08 EDT

I've no idea. I don't think so.

28th October 2006 @ 12:37 EDT

We're using nohup (http://209.85.129.104/search?q=cache:-CXhcg1gVgMJ:bama.ua.edu/cgi-bin/man-cgi%3Fnohup+man+nohup&hl=en&ct=clnk&cd=1&client=safari) to fire off a script in the background on a page request. This script then does all the work while the exec call immediately returns so the user doesn't have to wait. Since we're on Unix this is future-proof for us.

Best, Jan --

by Jan in .
28th October 2006 @ 12:49 EDT

forking works, but without coordination between your apache processes there is the potential to flood your box with those background processes.

You might also want that background processing to take place on a different box, leaving your web servers to serve the web.

If your operations folks are paranoid, your web server might be in a chroot or otherwise restricted environment that prevents you from spawning processes like this.

I'm not saying that your approach is bad (it obviously works well for you!), just highlighting some cases where it might not work for others.

28th October 2006 @ 13:41 EDT

One solution that will work in any database that supports triggers would be to setup a mail-queue table. Then make a trigger on inserts for this table who's job it would be to take the data and send an e-mail. The table can be as simple as

create table mail_queue { to varchar(255), subject varchar(255), body LONGTEXT, headers TEXT }

29th October 2006 @ 13:36 EDT

Thanks for pointing the possible weak spots of our approach! You're right, it works for us, but in other environments it might just not be feasable to fork.

Jan --

by Jan in .
6th November 2007 @ 11:29 EDT

Do you have a copy of the patch you can post? It doesn't seem to be committed as of 5.2.4. Also the PECL PDO extension hasn't been updated.

by Anish Mistry in .
3rd February @ 23:59 EDT

Track/vote on PHP bug #42614 if you're interested in seeing this become part of the standard PDO/PHP distribution. :-)

Wez mentioned he needed to work out a few things before committing the patch to CVS.

As a side note, a bug has been found in the interaction between PHP 5.2/PDO and PostgreSQL - see:

http://www.gavinroy.com/2008/2/3/ph...x-pdo-libpq-bug

Not specifically related to this patch, but if you're using PHP/PDO and PostgreSQL, probably something of which you should be aware.

by David Begley in .
29th February @ 10:54 EDT

hey, is there a chance to publish the PDO patch before it actually gets into official php release?

that'd be great:)

Post a comment