Balog, Krisztian, Kelly, Liadh ORCID: 0000-0003-1131-5238 and Schuth, Anne (2014) Head first: living labs for ad-hoc search evaluation. In: 23rd ACM International Conference on Information and Knowledge Management (CIKM 2014), 3-7 Nov 2014, Shanghai, China. ISBN 978-1-4503-2598-1
Abstract
The information retrieval (IR) community strives to make evaluation more centered on real users and their needs. The living labs evaluation paradigm, i.e., observing users in their natural task environments, offers great promise in this regard. Yet, progress in an academic setting has been limited. This paper presents the first
living labs for the IR community benchmarking campaign initiative, taking as test two use-cases: local domain search on a university website and product search on an e-commerce site. There are many challenges associated with this setting, including incorporating results from experimental search systems into live production
systems, and obtaining sufficiently many impressions from relatively low traffic sites. We propose that head queries can be used to generate result lists offline, which are then interleaved with results of the production system for live evaluation. An API is developed to orchestrate the communication between commercial parties and benchmark participants. This campaign acts to progress the living labs for IR evaluation methodology, and offers important insight into the role of living labs in this space.
Metadata
Item Type: | Conference or Workshop Item (Paper) |
---|---|
Event Type: | Conference |
Refereed: | Yes |
Uncontrolled Keywords: | Evaluation; Living Labs |
Subjects: | Computer Science > Information retrieval |
DCU Faculties and Centres: | Research Institutes and Centres > Centre for Next Generation Localisation (CNGL) DCU Faculties and Schools > Faculty of Engineering and Computing > School of Computing |
Published in: | Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management. . Association for Computing Machinery. ISBN 978-1-4503-2598-1 |
Publisher: | Association for Computing Machinery |
Official URL: | http://dx.doi.org/10.1145/2661829.2661962 |
Copyright Information: | © ACM, 2014. This is the author's version of the work. It is posted here by permission of ACM for your personal use. Not for redistribution. The definitive version was published in Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management http://dx.doi.org/10.1145/2661829.2661962 |
Use License: | This item is licensed under a Creative Commons Attribution-NonCommercial-Share Alike 3.0 License. View License |
ID Code: | 20131 |
Deposited On: | 26 Mar 2015 14:01 by Liadh Kelly . Last Modified 09 Nov 2018 13:49 |
Documents
Full text available as:
Preview |
PDF
- Requires a PDF viewer such as GSview, Xpdf or Adobe Acrobat Reader
193kB |
Metrics
Altmetric Badge
Dimensions Badge
Downloads
Downloads
Downloads per month over past year
Archive Staff Only: edit this record