Sparrow is a really fast lightweight queue written in Ruby that speaks memcache. That means you can use Sparrow with any memcached client library (Ruby or otherwise).

Basic tests shows that Sparrow processes messages at a rate of 1000 per second (depending on how you store messages). The load Sparrow can cope with increases exponentially as you add to the cluster. Sparrow also takes advantage of eventmachine, which uses a non-blocking io, offering great performance.

Sparrow's speed greatly increases when using the included client.

Sparrow comes with built in support for daemonization and clustering. Also included are example libraries and clients. For example:

require 'memcache'
m = MemCache.new('127.0.0.1:11212')
m['queue_name'] = '1' # Publish to queue
m['queue_name']       #=> 1 Pull next msg from queue
m['queue_name']       #=> nil
m.delete('queue_name) # Delete queue

# or using the included client:

class MyQueue < MQ3::Queue
  def on_message(args)
    puts "Received msg with args: #{args.inspect}"
  end
end

MyQueue.servers = [
  MQ3::Protocols::Memcache.new({:host => '127.0.0.1', :port => 11212, :weight => 1})
]
MyQueue.publish('test msg')
MyQueue.run

Messages are deleted as soon as they're read and the order you add messages to the queue probably won't be the same order when they're removed.

Additional memcached commands that are supported are:

flush_all # Deletes all queues
version
quit

The memcached commands 'add', and 'replace' just call 'set'.

Command line options are:

Usage: sparrow [-b path] [-t type] [-h host] [-p port] [-P file]
               [-d] [-k port] [-l file] [-e]
       sparrow --help
       sparrow --version


Configuration:
    -b, --base PATH           Path to queue data store.
                              (default: /var/spool/sparrow)
    -t, --type QUEUE_TYPE     Type of queue (disk/memory/sqlite).
                              (default: disk)

Network:
    -h, --host HOST           Specify host
                              (default: 0.0.0.0)
    -p, --port PORT           Specify port
                              (default: 11212)

Daemonization:
    -P, --pid FILE            save PID in FILE when using -d option.
                              (default: /var/run/sparrow.log)
    -d, --daemon              Daemonize mode
    -k, --kill PORT           Kill specified running daemons - leave blank to kill all.

Logging:
    -l, --log [FILE]          Path to print debugging information.
    -e, --debug               Run in debug mode
                              (default: false)

Miscellaneous:
    -?, --help                Display this usage information.
    -v, --version             Display version

The only prerequisite is eventmachine, but you'll also need the sqlite3-ruby gem if you're using Sqlite to store messages. The daemonization won't work on Windows.

Check out the code: svn checkout http://sparrow.googlecode.com/svn/trunk/ sparrow

Install the gem: sudo gem install sparrow

Client

Also included is a Sparrow client with advanced clustering capabilities and fallbacks. If the Sparrow servers aren't available, queuing can fallback to Amazon SQS (you'll need the SQS gem if you're using this).

class MyQueue < MQ3::Queue
  def on_message(args)
    puts "Received msg with args: #{args.inspect}"
  end
end

SQS.access_key_id = 'YOURACCESSKEYID'
SQS.secret_access_key = 'YOURSECRETACCESSKEY'
   
servers = [
   MQ3::Protocols::Memcache.new({:host => 'localhost', :port => 11211, :weight => 1}),
   MQ3::Protocols::Memcache.new({:host => 'localhost', :port => 11212, :weight => 1}),
   MQ3::Protocols::Memcache.new({:host => 'localhost', :port => 11213, :weight => 1}),
   MQ3::Protocols::Memcache.new({:host => 'localhost', :port => 11214, :weight => 1}),
   MQ3::Protocols::SQS.new({:weight => 0})
]


MyQueue.servers = servers

MyQueue.publish 'test'

MyQueue.run

Sparrow was inspired by Twitter's Starling