zoie

Zoie issue tracking has been moved to: http://linkedin.jira.com/browse/ZOIE

Zoie wiki has been moved to: http://linkedin.jira.com/wiki/display/ZOIE/Home

Zoie source control has been moved to: http://github.com/javasoze/zoie

Zoie is a real-time search and indexing system built on Apache Lucene.

News: Zoie 2.0.0-rc3 is released (2/17/2010) - Requires Lucene 2.9.x or Lucene 3.0.

Originally developed at LinkedIn.com.

Donated by LinkedIn.com on July 19, 2008.

Zoie is a mature open source project and has been deployed in a real-time large-scale consumer website: LinkedIn.com handling millions of searches as well as hundreds of thousands of updates daily.

All Zoie releases have gone through extensive functional and performance testing by LinkedIn before made public.

In a real-time search/indexing system, a document is made available as soon as it is added to the index. This functionality is especially important to time-sensitive information such as news, job openings, tweets etc.

This poses the following challenges which Zoie addresses:

Additions of documents must be made available to searchers immediately
Indexing must not affect search performance
Additions of documents must not fragment the index (which hurts search performance)
Deletes and/or updates of documents must not affect search performance.
...

Additional Zoie features:

fast lucene docid to uid mapping
fast uid to lucene docid mapping (reverse id mapping)
Live updates
custom MergePolicy to handle realtime updates
partial delete expunge for enhancing search performance without full optimize
balanced index segment management
full jmx console for indexing management/monitoring
...

Architecture Diagram:

YourKit is kindly supporting open source projects with its full-featured Java Profiler. YourKit, LLC is the creator of innovative and intelligent tools for profiling Java and .NET applications. Take a look at YourKit's leading software products: YourKit .NET Profiler and YourKit Java Profiler.