Examples of topic indexing with Maui

• Examples of topic indexing with Maui
• Automatic tagging with Maui
• Assignment of Wikipedia terms to Computer Science articles
• Assignment of Agrovoc terms to agricultural documents
• English agricultural documents
• French and Spanish agricultural documents
• Indexing with Medical Subject Headings
• Subject indexing with High Energy Physics thesaurus terms

All examples are produced using data sets available for download under Resources and MultiplyIndexedData.

Automatic tagging with Maui

We run a test on 180 documents from CiteULike. For each document, tags on which at least two human taggers agreed were considered as correct. All other tags as incorrect. Maui achieved F-Measure of nearly 50%, i.e. a half of the top 10 automatically extracted tags were correct.

Here are three examples of different performance levels. Numbers in brackets indicate how many people agreed on this tag. Bold indicates a correct term. Italic indicates that Maui matched a tag assigned by one person.

Excellent performance (F-Measure 80%). Different time courses of learning-related activity in the prefrontal cortex and striatum

Average performance (F-Measure 46%). Global and regional drivers of accelerating CO2 emissions

Poor performance (F-Measure 0%). Drought sensitivity shapes species distribution patterns in tropical forests.

Assignment of Wikipedia terms to Computer Science articles

In this experiment, 20 computer science technical reports were used. Each document received topics from 15 teams of graduate students. Multiple data set was used to compute Maui's consistency with the consistency of people with each other. On average Maui outperforms 8 of the human teams.

Consistency analysis is used to show three different levels of Maui's performance. Numbers in brackets indicate how many human teams chose that topic.

Excellent performance. Maui: 52%, people 42%

Document Occam's razor: The cutting edge for parser technology

Average performance. Maui: 30%, people 53%

Document A Safe, Efficient Regression Test Selection Technique

Poor performance. Maui: 17%, people 17%

Document Cone trees in the UGA graphics system

Assignment of Agrovoc terms to agricultural documents

These experiments resulted from the collaboration with the Food and Agriculture Organization (FAO) of the United Nations. Each document is automatically indexed with terms from the domain-specific thesaurus Agrovoc. The results are compared to those assigned by professionals.

English agricultural documents

Six professional indexers at FAO indexed 30 documents each. Using consistency measure, we computed the agreement among the indexers and their agreement with Maui. Maui's consistency is only slightly worse than the consistency of professionals.

Here are examples of three performance levels that Maui achieved:

Excellent performance. Maui 51% vs. Indexers 40%

Document The dynamics of sanitary and technical requirement assisting the poor

Average performance. Maui 29% vs. Indexers 35%

Document Climate change and the forest sector

Poor performance. Maui 7% vs. Indexers 35%

Document Phosphorus limitation of microbial processes in tropical forests

French and Spanish agricultural documents

After training on 66 French and 46 Spanish documents, no modification to code other than a new stemmer, stopwords list and encoding. Matching terms are shown in bold.

French document Relations terres-eau dans les bassins versant ruraux.

Spanish document Evaluación de los recursos forestales mundiales 2000 .

Indexing with Medical Subject Headings

After training on 450 documents from PubMed, Maui assigned MeSH terms like the following.

Document Determining lifestyle correlates of body mass index using multilevel analyses.

Subject indexing with High Energy Physics thesaurus terms

Example document from CERN Document server Two-loop electroweak corrections to Higgs production at hadron colliders.