Sparrow is a really fast lightweight queue written in Ruby that speaks memcache. That means you can use Sparrow with any memcached client library (Ruby or otherwise).

Sparrow keeps messages in memory, but persists them to disk, using Sqlite, when the queue is shutdown.

Basic tests shows that Sparrow processes messages at a rate of around 1660 per second. The load Sparrow can cope with increases exponentially as you add to the cluster. Sparrow also takes advantage of eventmachine, which uses a non-blocking io, offering great performance.

Sparrow's speed greatly increases when using the included client.

Sparrow comes with built in support for daemonization and clustering. Also included are example libraries and clients. For example:

require 'memcache'
m = MemCache.new('127.0.0.1:11212')
m['queue_name'] = '1' # Publish to queue
m['queue_name']       #=> 1 Pull next msg from queue
m['queue_name']       #=> nil
m.delete('queue_name) # Delete queue

Messages are deleted as soon as they're read and the order you add messages to the queue probably won't be the same order when they're removed.

Additional memcached commands that are supported are:

flush_all # Deletes all queues
version
quit

The memcached commands 'add', and 'replace' just call 'set'.

Command line options are:

Usage: sparrow [-b path] [-h host] [-p port] [-P file]
               [-d] [-k port] [-l file] [-e]
       sparrow --help
       sparrow --version


Configuration:
    -b, --base PATH           Path to queue data store.
                              (default: /var/spool/sparrow)

Network:
    -h, --host HOST           Specify host
                              (default: 0.0.0.0)
    -p, --port PORT           Specify port
                              (default: 11212)

Daemonization:
    -P, --pid FILE            save PID in FILE when using -d option.
                              (default: /var/run/sparrow.log)
    -d, --daemon              Daemonize mode
    -k, --kill PORT           Kill specified running daemons - leave blank to kill all.

Logging:
    -l, --log [FILE]          Path to print debugging information.
    -e, --debug               Run in debug mode
                              (default: false)

Miscellaneous:
    -?, --help                Display this usage information.
    -v, --version             Display version

Prerequisites are eventmachine and the sqlite3-ruby gem.

The daemonization won't work on Windows.

Check out the code: svn checkout http://sparrow.googlecode.com/svn/trunk/ sparrow

Install the gem: sudo gem install sparrow

Client

Also included is a Sparrow client with advanced clustering capabilities and fallbacks. If the Sparrow servers aren't available, queuing can fallback to Amazon SQS (you'll need the SQS gem if you're using this).

class MyQueue < MQueue::Queue
  def on_message(args)
    puts "Received msg with args: #{args.inspect}"
  end
end

SQS.access_key_id = 'YOURACCESSKEYID'
SQS.secret_access_key = 'YOURSECRETACCESSKEY'
   
servers = [
   MQueue::Protocols::Sparrow.new({:host => 'localhost', :port => 11212, :weight => 1}),
   MQueue::Protocols::SQS.new({:weight => 0})
]

MyQueue.servers = servers
MyQueue.publish 'test'
MyQueue.run

MyQueue.publish 'test'

MyQueue.run

Sparrow was inspired by Twitter's Starling