ISSN:
1572-8412
Keywords:
evaluation
;
ambiguity resolution
;
WSD
;
inter-annotator agreement
Source:
Springer Online Journal Archives 1860-2000
Topics:
Computer Science
,
Media Resources and Communication Sciences, Journalism
Notes:
Abstract We present methods for evaluating human and automatictaggers that extend current practice in three ways. First, we show howto evaluate taggers that assign multiple tags to each test instance,even if they do not assign probabilities. Second, we show how toaccommodate a common property of manually constructed ``gold standards''that are typically used for objective evaluation, namely that there isoften more than one correct answer. Third, we show how to measureperformance when the set of possible tags is tree-structured in an IS-Ahierarchy. To illustrate how our methods can be used to measureinter-annotator agreement, we show how to compute the kappa coefficientover hierarchical tag sets.
Type of Medium:
Electronic Resource
URL:
http://dx.doi.org/10.1023/A:1002402902356
Permalink
