A strategy for evaluating search of “Real” personal
information archives
Jones, Gareth J.F.ORCID: 0000-0003-2923-8365 and Chen, Yi
(2011)
A strategy for evaluating search of “Real” personal
information archives.
In: The Evaluating Personal Search Workshop at ECIR 2011, 18-21 April 2011, Dublin, Ireland.
Personal information archives (PIAs) can include materials from many sources, e.g. desktop and laptop computers, mobile phones, etc. Evaluation of personal search over these collections is problematic for reasons relating to the personal and private nature of the data and associated information needs and measuring system response effectiveness. Conventional information retrieval (IR) evaluation involving use of Cranfield type test collections to establish retrieval effectiveness and laboratory testing of interactive search behaviour have to be re-thought in this situation. One key issue is that personal data and information needs are very different to search of more public third party datasets used in most existing evaluations. Related to this, understanding the issues of how users interact with a search system for their personal data is important in developing search in this area on a well grounded basis. In this proposal we suggest an
alternative IR evaluation strategy which preserves privacy
of user data and enables evaluation of both the accuracy of
search and exploration of interactive search behaviour. The
general strategy is that instead of a common search dataset
being distributed to participants, we suggest distributing
standard expandable personal data collection, indexing and
search tools to non-intrusively collect data from participants conducting search tasks over their own data collections on their own machines, and then performing local evaluation of individual results before central agregation.