title: New metrics for meaningful evaluation of informally structured speech retrieval
creator: Eskevich, Maria
creator: Magdy, Walid
creator: Jones, Gareth J.F.
subject: Multimedia systems
subject: Information retrieval
subject: Speech retrieval; informally structured speech; evaluation metrics
description: Search effectiveness for tasks where the retrieval units are
clearly defined documents is generally evaluated using standard measures such as mean average precision (MAP). However, many practical speech search tasks focus on content within large spoken files lacking defined structure. These data must be segmented into smaller units for search which may only partially overlap with relevant material. We introduce two new metrics for the evaluation of search effectiveness for informally structured speech data: mean average segment precision (MASP) which measures retrieval performance in terms of both content segmentation and ranking with respect to relevance; and mean average segment distance-weighted precision (MASDWP) which takes into account the distance between the start of the relevant segment and the retrieved segment. We demonstrate the eectiveness of these new metrics on a retrieval test collection based on the AMI meeting corpus.
publisher: Springer
date: 2012-04-01
type: Conference or Workshop Item
type: PeerReviewed
format: application/pdf
language: en
relation: http://doras.dcu.ie/16900/1/full_paper_ECIR_2012.pdf
relation: http://dx.doi.org/10.1007/978-3-642-28997-2_15 
identifier: http://doras.dcu.ie/16900/
source: Eskevich, Maria and Magdy, Walid and Jones, Gareth J.F. (2012) New metrics for meaningful evaluation of informally structured speech retrieval. In: 34th European Conference on Information Retrieval Research,, 1-5 Apr 2012, Barcelona, Spain. ISBN 978-3-642-28997-2
rights: © 2012 Springer-Verlag.
The original publication is available at www.springerlink.com
rights: This item is licensed under a Creative Commons Attribution-NonCommercial-Share Alike 3.0 License.