My favorites | Sign in
Project Home Downloads Wiki Issues Source
Project Information
Members
Featured
Downloads
Wiki pages
Links

Notice: The workerpool project is migrating to Github. Please refer to it here: https://github.com/shazow/workerpool


Performing tasks in many threads made fun!

This module facilitates distributing simple operations into jobs that are sent to worker threads, maintained by a pool object.

It consists of these components:

  1. Jobs, which are single units of work that need to be performed.
  2. Workers, who grab jobs from a queue and perform them.
  3. Worker pool, which keeps track of workers and the job queue.

Getting Started

The best place to start for now is to read the code and look at the examples in the unit tests (located under trunk/test/) and the sample uses (located under trunk/samples/). Documentation contributions are welcome! What have you accomplished with the workerpool?

Tutorials:

  • MassDownloader - How to write a simple multi-threaded mass downloader in under 10 lines of code.

Status

There's good work being done on a native Python multiprocessing module. The functionality has a lot of overlap with workerpool. Worth having a look at!

News

Todo

  • Add more complete usage examples and wiki tutorials.
  • More thorough unit testing
  • Finalize the API for a 1.0 release

Using workerpool

  • s3funnel - Multithreaded tool for performing operations on Amazon's S3
  • Are you using it? Let me know!

Credit

This module was originally developed during my work at Idée Inc. Big thanks to Idée for letting me open source it!

Powered by Google Project Hosting