DirectoryQueue

A Python package providing an abstraction around a filesystem-based queue. All queue operations are atomic as they rely on directory operations, so it is commonly used as a persistent queue between services.

Example:

  >>> from directory_queue.directory_queue import DirectoryQueue
  >>> from directory_queue.generic_queue_item import GenericQueueItem
  >>> import os
  >>> 
  >>> qdir = '/tmp/queue'
  >>> os.mkdir(qdir)
  >>> dq = DirectoryQueue(qdir, GenericQueueItem)
  >>> 
  >>> item1 = dq.newQueueItem('item1')
  >>> file(item1.dataFileName(), 'w').write('some data')
  >>> item1.writeMetadata({'what': 'anything', 'foo': 'bar'})
  >>> dq.itemReady(item1)
  >>> 
  >>> ready_item1 = dq.getNext()
  >>> md = ready_item1.readMetadata()
  >>> md
  {'what': 'anything', 'foo': 'bar'}
  >>> data = file(ready_item1.dataFileName(), 'r').read()
  >>> data
  'some data'
  >>> dq.itemDone(ready_item1)

DirectoryQueueService

DirectoryQueueService is a Twisted IService that polls a DirectoryQueue and fires callbacks to process new queue items when they become available.

The DirectoryQueueService supports processing items concurrently, up to the concurrency level specified by the constructor parameter.

Note that the user-supplied callbacks must take care of the details of running asynchronously for actual concurrency to occur. See itemReadyCallback method.