Magdy, Walid and Jones, Gareth J.F.ORCID: 0000-0003-2923-8365
(2010)
A new metric for patent retrieval evaluation.
In: the 1st International Workshop on Advances in Patent Information Retrieval (AsPIRe'10) at 32nd European Conference on Information Retrieval (ECIR 2010), 28 March 2010, Milton Keynes, U.K..
Patent retrieval is generally considered to be a recall-oriented information retrieval task that is growing in importance. Despite this fact, precision based scores such as mean average precision (MAP) remain the primary evaluation measures for patent retrieval. Our study examines different evaluation measures for the recall-oriented patent retrieval task and shows the limitations
of the current scores in comparing different IR systems for this task. We introduce PRES, a novel evaluation metric for this type of application taking account of recall and user search effort. The behaviour of PRES is demonstrated on 48 runs from the CLEF-IP 2009 patent retrieval track. A full analysis of the performance of PRES shows its suitability for measuring the retrieval effectiveness of systems from a recall focused perspective taking into account the expected search effort of patent searchers.