ISSN:
1572-8412
Keywords:
evaluation
;
SENSEVAL
;
word sense disambiguation
Source:
Springer Online Journal Archives 1860-2000
Topics:
Computer Science
,
Media Resources and Communication Sciences, Journalism
Notes:
Abstract Senseval was the first open, community-based evaluation exercisefor Word Sense Disambiguation programs. It adopted the quantitativeapproach to evaluation developed in MUC and other ARPA evaluationexercises. It took place in 1998. In this paper we describe thestructure, organisation and results of the SENSEVAL exercise forEnglish. We present and defend various design choices for theexercise, describe the data and gold-standard preparation, considerissues of scoring strategies and baselines, and present the resultsfor the 18 participating systems. The exercise identifies thestate-of-the-art for fine-grained word sense disambiguation, wheretraining data is available, as 74–78% correct, with a number ofalgorithms approaching this level of performance. For systems thatdid not assume the availability of training data, performance wasmarkedly lower and also more variable. Human inter-tagger agreementwas high, with the gold standard taggings being around 95%replicable.
Type of Medium:
Electronic Resource
URL:
http://dx.doi.org/10.1023/A:1002693207386
Permalink
