zoie


real-time search and indexing system built on Apache Lucene

New Zoie Home: http://javasoze.github.com/zoie

Zoie issue tracking has been moved to: http://linkedin.jira.com/browse/ZOIE

Zoie wiki has been moved to: http://linkedin.jira.com/wiki/display/ZOIE/Home

Zoie source control has been moved to: http://github.com/javasoze/zoie

Zoie is a real-time search and indexing system built on Apache Lucene.

News: Zoie 2.0.0-rc3 is released (2/17/2010) - Requires Lucene 2.9.x or Lucene 3.0.

Originally developed at LinkedIn.com.

Donated by LinkedIn.com on July 19, 2008.

Zoie is a mature open source project and has been deployed in a real-time large-scale consumer website: LinkedIn.com handling millions of searches as well as hundreds of thousands of updates daily.

All Zoie releases have gone through extensive functional and performance testing by LinkedIn before made public.


In a real-time search/indexing system, a document is made available as soon as it is added to the index. This functionality is especially important to time-sensitive information such as news, job openings, tweets etc.

This poses the following challenges which Zoie addresses:

  • Additions of documents must be made available to searchers immediately
  • Indexing must not affect search performance
  • Additions of documents must not fragment the index (which hurts search performance)
  • Deletes and/or updates of documents must not affect search performance.
  • ...

Additional Zoie features:

  • fast lucene docid to uid mapping
  • fast uid to lucene docid mapping (reverse id mapping)
  • Live updates
  • custom MergePolicy to handle realtime updates
  • partial delete expunge for enhancing search performance without full optimize
  • balanced index segment management
  • full jmx console for indexing management/monitoring
  • ...

Architecture Diagram:

http://sites.google.com/site/johnwang/geekstuff/zoie-graph.jpg


YourKit is kindly supporting open source projects with its full-featured Java Profiler. YourKit, LLC is the creator of innovative and intelligent tools for profiling Java and .NET applications. Take a look at YourKit's leading software products: http://www.yourkit.com/dotnet/download/index.jsp'>YourKit .NET Profiler and http://www.yourkit.com/java/profiler/index.jsp'>YourKit Java Profiler.

Project Information

Labels:
java lucene search zoie realtime indexing opensource