Browse DORAS
Browse Theses
Latest Additions
Creative Commons License
Except where otherwise noted, content on this site is licensed for use under a:

A study of inter-annotator agreement for opinion retrieval

Bermingham, Adam and Smeaton, Alan F. (2009) A study of inter-annotator agreement for opinion retrieval. In: SIGIR 2009 - The 32nd Annual ACM SIGIR Conference , 20-22 July 2009, Boston, USA. ISBN 978-1-60558-483-6

Full text available as:

PDF - Requires a PDF viewer such as GSview, Xpdf or Adobe Acrobat Reader


Evaluation of sentiment analysis, like large-scale IR evalu- ation, relies on the accuracy of human assessors to create judgments. Subjectivity in judgments is a problem for rel- evance assessment and even more so in the case of senti- ment annotations. In this study we examine the degree to which assessors agree upon sentence-level sentiment anno- tation. We show that inter-assessor agreement is not con- tingent on document length or frequency of sentiment but correlates positively with automated opinion retrieval per- formance. We also examine the individual annotation cate- gories to determine which categories pose most di±culty for annotators.

Item Type:Conference or Workshop Item (Poster)
Event Type:Conference
Subjects:Computer Science > Computational linguistics
Computer Science > Machine learning
Computer Science > Information retrieval
DCU Faculties and Centres:DCU Faculties and Schools > Faculty of Engineering and Computing > School of Computing
Research Initiatives and Centres > CLARITY: The Centre for Sensor Web Technologies
Published in:SIGIR 2009: Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval. . Association for Computing Machinery. ISBN 978-1-60558-483-6
Publisher:Association for Computing Machinery
Official URL:
Copyright Information:Copyright 2009 the authors
Use License:This item is licensed under a Creative Commons Attribution-NonCommercial-Share Alike 3.0 License. View License
Funders:Science Foundation Ireland, SFI 07/CE/I1147
ID Code:14779
Deposited On:05 Aug 2009 15:48 by Adam Bermingham. Last Modified 12 Jan 2017 12:25

Download statistics

Archive Staff Only: edit this record