DIMPLE

This codebase includes: * DIMPLE (DIversity-aware Metric for Pattern Learning Experiment) implementation that evaluates paraphrase patterns. * Data loader (edu.cmu.lti.dimple.data.*PairLoader) for (RTE, MS Paraphrase, Complex QA evaluation) datasets which could be reused in other projects. * Experiment launcher to reproduce results in (Shima & Mitamura, 2011).

Recommended Environment

• JDK 6 or later
• Eclipse

Instructions

• Download the codebase from Downloads page

• Download data from the following websites and put them under the "resources" directory.

  • MSRPC by MSR
  • RTE (RTE1-4) by PASCAL/TAC
  • Complex QA Evaluation (2005 other, 2005 relation, 2006 ciQA, 2006 other, 2007 ciQA, 2007 other) by TREC
  • paraphrase-eval by USC ISI

Directory structure: /** resources/ |-- mavuno-annotations | |-- lenient-filtered.annotations | `-- strict-filtered.annotations |-- mavuno-paraphrases | |-- bcb-s.results | |-- bcb.results | : : : |-- msrpc | |-- msr_paraphrase_test.txt | `-- msr_paraphrase_train.txt |-- qa | |-- 2005_other | | |-- 05.nuggets.other.answers.txt | | `-- other | | |-- RUN-100.otherjudged | | |-- RUN-101.otherjudged | | : : : | |-- 2005_relation | | |-- 05.nuggets.list.txt | | `-- relate.results | | |-- RUN-1.relatejudged | | |-- RUN-10.relatejudged | | : : : | |-- 2006_ciqa | | |-- 06.ciqa_answers.txt | | `-- ciqa_judgments | | |-- RUN-1.ciqajudged | | |-- RUN-10.ciqajudged | | : : : | |-- 2006_other | | |-- 06.other_answers.txt | | `-- other_judgments | | |-- RUN-100.otherjudged | | |-- RUN-101.otherjudged | | : : : | |-- 2007_ciqa | | |-- 2007_ciqa_judgments | | | |-- RUN-22-1.ciqajudged | | | |-- RUN-22-2.ciqajudged | | | : : : | | |-- ciqa_nugget_pyramid.txt | | `-- ciqa_nuggets.txt | `-- 2007_other | |-- 2007_other_judgments | | |-- RUN-1-1.other_judged | | |-- RUN-1-2.other_judged | | : : : | `-- other_nugget_pyramid.txt |-- rte | |-- RTE1_dev.xml | |-- RTE1_dev2.xml | |-- RTE1_test.xml | |-- RTE2_dev.xml | |-- RTE2_test.xml | |-- RTE3-TEST-GOLD.xml | |-- RTE3_pairs_dev-set-final.xml | |-- RTE4_TEST-SET_GOLD.xml | `-- rte.dtd `-- terms `-- verb_list_textinfer2011.txt **/

• If you use eclipse, use the launch file launches\TextInfer2011Experiment.launch to run experiments at (Shima & Mitamura, 2011).

Reference

• Shima, Hideki and Teruko Mitamura. 2011. Diversity-aware Evaluation for Paraphrase Patterns. In Proceedings of TextInfer 2011: The EMNLP 2011 Workshop on Textual Entailment.