RITE SDK

Introduction

RITE SDK provides a Java framework for rapidly building a Textual Entailment recognition system especially toward participating in the NTCIR-9 RITE evaluation task. RITE SDK comes with a sample code, so you can rapidly build a working system by modifying it. The framework also provides evaluation capability with a confusion-matrix error analysis.

What's new

Version 1.12 (Sep 28, 2011) - initialization and on-complete methods are available (optional). Batch evaluation is possible with more detailed evaluation logs.
Version 1.11 (Sep 15, 2011) - Bugfix: evaluation result not shown from the run launched from MainClass.
Version 1.10 (Sep 14, 2011) - Abstract RITE system work with input file without gold label; Evaluation available given a run result and gold standard file.

Requirement

JDK 6 or later
(recommended) Eclipse + Subclipse

In this documentation, we assume you have eclipse environments.

Instructions to run sample codes

Obtain the framework project (edu.cmu.lti.ritesdk and a sample implementation edu.cmu.lti.ritesdk.sample)

Download the archive from the Downloads page. Then import codes into eclipse workspace: File > Import > General > Existing Projects into Workspace.
Or, use Subclipse to check out: File > Import > SVN > Checkout Projects from SVN. Specify the repository at http://rite-sdk.googlecode.com/svn/trunk/ with anonymous-checkout.

If you are simply interested in browsing the source code before obtaining the entire codebase, go to the web-based repository browser.

Download the input data from RITE website and put them under edu.cmu.lti.ritesdk.sample/data (see a screenshot below).

If you store them in another location, modify the -i argument of the launch files.

Right click on the launch files at edu.cmu.lti.ritesdk.sample/launches/* and click "Run As" to run the sample systems.

For Simplified/Traditional Chinese, we only delivered MC development data. So please create a BC data through the following conversion: replace I&C labels with N; replace F&B labels with Y; and replace R labels with Y with t1, t2 switched.

Verify the results (e.g. accuracy, confusion matrix) in the console.

Code base outline

Output from sample code

Sample codes

edu.cmu.lti.ritesdk.sample project provides the following sample implementations.

Class Name	Description
`edu.cmu.lti.ritesdk.sample.CharOverlapBasedSystem`	Decides on a label based on the portion of t2 contained in t1 in character-level.
`edu.cmu.lti.ritesdk.sample.RandomMCSystem`	Toy implementation for MC subtask that randomly returns labels.
`edu.cmu.lti.ritesdk.sample.RandomMultiEngineSystem`	Composite RITE system for BC subtask; Weighted multi-engine design allows you to combine multiple RITE systems.

Class and Arguments

`edu.cmu.lti.ritesdk.MainClass`

The main class for invoking the RITE system, which takes the following arguments.

usage: help
 -h,--help           print this message and exit
 -i,--input <arg>    input file
 -s,--system <arg>   system class name
 -t,--task <arg>     task name (BC or MC)

Example:

-t bc -i ./data/RITE1_JA_test_bc.txt -s edu.cmu.lti.ritesdk.sample.CharOverlapBasedSystem

`edu.cmu.lti.ritesdk.MainClassForEvaluation`

The main class for evaluating a run result, which takes the following arguments.

usage: help
 -h,--help                  print this message and exit
 -i,--gold standard <arg>   gold standard file
 -r,--run result <arg>      run result file, or directory with multiple result files
 -t,--task <arg>            task name (BC or MC)

Example:

-t bc -i ./data/RITE1_JA_test_bc.txt -r ./target/RITE1-LTI-JA-BC-01.txt

For more examples, see the actual arguments in the launch configurations.

UML

In case you are interested in more details of the software, we provide UML diagrams below. Some minor details are simplified or omitted.

Class Diagram

Sequence Diagram

Reference

Shima, Hideki, Hiroshi Kanayama, Cheng-Wei Lee, Chuan-Jie Lin, Teruko Mitamura, Yusuke Miyao, Shuming Shi, and Koichi Takeda. 2011. "Overview of NTCIR-9 RITE: Recognizing Inference in TExt". In Proceedings of NTCIR-9 Workshop, Japan.