A study of inter-annotator agreement for opinion retrieval
Bermingham, Adam and Smeaton, Alan F.
(2009)
A study of inter-annotator agreement for opinion retrieval.
In: SIGIR 2009 - The 32nd Annual ACM SIGIR Conference, 20-22 July 2009, Boston, USA.
ISBN 978-1-60558-483-6
Evaluation of sentiment analysis, like large-scale IR evalu-
ation, relies on the accuracy of human assessors to create
judgments. Subjectivity in judgments is a problem for rel-
evance assessment and even more so in the case of senti-
ment annotations. In this study we examine the degree to
which assessors agree upon sentence-level sentiment anno-
tation. We show that inter-assessor agreement is not con-
tingent on document length or frequency of sentiment but
correlates positively with automated opinion retrieval per-
formance. We also examine the individual annotation cate-
gories to determine which categories pose most di±culty for
annotators.
SIGIR 2009: Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval.
.
Association for Computing Machinery. ISBN 978-1-60558-483-6