My favorites | Sign in
Project Home Downloads Wiki Issues Source
Project Information
Members
Links

Register a function as accepting ticket requests, and queue up a model instance as needing servicing. later on, your model instance will be popped out of the database and serviced, then removed from the queue. This sounds similar to RabbitMQ, or one of those other message queues, and it is somewhat. For the general case, though, this is far less efficient, requiring database writes a-plenty and lots of reads.

But is much more efficient for our use case, where

  • the queued tasks are very expensive (seconds or occasionally minutes to process a single record e.g. geocoding by underfunded government service such as https://six.maps.nsw.gov.au/wps/portal/ )
  • AND the queued tasks use rate-limited web services, so we need to avoid duplicates to get the most out of our quota
  • BUT but are idempotent (running the function once is the same as running it many times).

In our case, the overhead of doing this in a database is worth the ability to squash multiple function calls into one. you might think of this as complementary to a conventional queue. In other ways it's less sophisticated than a conventional queue - no sophisticated exchange routing, etc.

created for http://about.nsw.gov.au/

Powered by Google Project Hosting