Browse DORAS
Browse Theses
Search
Latest Additions
Creative Commons License
Except where otherwise noted, content on this site is licensed for use under a:

PRES: A score metric for evaluating recall-oriented information retrieval applications

Magdy, Walid and Jones, Gareth J.F. (2010) PRES: A score metric for evaluating recall-oriented information retrieval applications. In: The 33rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2010), 19-23 Jul 2010, Geneva, Switzerland. ISBN 978-1-4503-0153-4

Full text available as:

[img]
Preview
PDF - Requires a PDF viewer such as GSview, Xpdf or Adobe Acrobat Reader
646Kb

Abstract

Information retrieval (IR) evaluation scores are generally designed to measure the effectiveness with which relevant documents are identified and retrieved. Many scores have been proposed for this purpose over the years. These have primarily focused on aspects of precision and recall, and while these are often discussed with equal importance, in practice most attention has been given to precision focused metrics. Even for recalloriented IR tasks of growing importance, such as patent retrieval, these precision based scores remain the primary evaluation measures. Our study examines different evaluation measures for a recall-oriented patent retrieval task and demonstrates the limitations of the current scores in comparing different IR systems for this task. We introduce PRES, a novel evaluation metric for this type of application taking account of recall and the user’s search effort. The behaviour of PRES is demonstrated on 48 runs from the CLEF-IP 2009 patent retrieval track. A full analysis of the performance of PRES shows its suitability for measuring the retrieval effectiveness of systems from a recall focused perspective taking into account the user’s expected search effort.

Item Type:Conference or Workshop Item (Paper)
Event Type:Conference
Refereed:Yes
Uncontrolled Keywords:PRES; Recall-Oriented Information Retrieval; Patent Retrieval; Evaluation Metric
Subjects:Computer Science > Information retrieval
DCU Faculties and Centres:Research Initiatives and Centres > Centre for Next Generation Localisation (CNGL)
DCU Faculties and Schools > Faculty of Engineering and Computing > School of Computing
Published in:SIGIR 2010 Proceedings - 33rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. . Association for Computing Machinery. ISBN 978-1-4503-0153-4
Publisher:Association for Computing Machinery
Official URL:http://portal.acm.org/ft_gateway.cfm?id=1835551&type=pdf&CFID=20871752&CFTOKEN=66848416
Copyright Information:© ACM, 2010. This is the author's version of the work. It is posted here by permission of ACM for your personal use. Not for redistribution. The definitive version was published in SIGIR 2010 Proceedings - 33rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 2010 http://doi.acm.org/10.1145/1835449.1835551
Use License:This item is licensed under a Creative Commons Attribution-NonCommercial-Share Alike 3.0 License. View License
ID Code:16180
Deposited On:22 Jul 2011 11:55 by Shane Harper. Last Modified 22 Jul 2011 11:55

Download statistics

Archive Staff Only: edit this record