Login (DCU Staff Only)
Login (DCU Staff Only)

DORAS | DCU Research Repository

Explore open access research and scholarly works from DCU

Advanced Search

Pushing the Boundaries of Lifelog Retrieval Systems with Question Answering Techniques

Tran, Ly-Duyen orcid logoORCID: 0000-0002-9597-1832 (2024) Pushing the Boundaries of Lifelog Retrieval Systems with Question Answering Techniques. PhD thesis, Dublin City University.

Abstract
Lifelogging, referring to the continuous capturing of personal experiences using digital devices such as wearable cameras and smart sensors, could be a valuable memory enhance- ment and provide great insights into how an individual lives their life. This thesis focuses on the novel application of question answering (QA) within the context of lifelogging, aiming to develop an advanced interactive lifelog retrieval system that supports answering questions based on lifelog data. To achieve this objective, this research addresses several key components. First, a novel lifelog QA dataset, named LLQA, was created, consisting of over 15,000 multiple-choice and yes-no questions regarding data from lifelog segmenta- tions. I evaluated different existing QA models on their suitability and capability to an- swer lifelog questions and compared their accuracies to the human baseline of 85.46%. The evaluation results recognised the efficiency of leveraging large pre-trained video-language models, achieving an accuracy of 72.43%, as opposed to constructing custom-built LLQA models, which achieved 71.23%. Next, I designed and continually enhanced MyScéal, a state-of-the-art interactive lifelog retrieval system that supports the user to efficiently retrieve relevant data in response to search queries and lifelog questions. This effort cul- minated in MyScéal’s success as the winning system in three consecutive iterations of Lifelog Search Challenges, underscoring its strengths in supporting the user to quickly locate items of interest from a conventional multimodal lifelog. Finally, a novel lifelog QA pipeline was proposed to seamlessly integrate QA models into existing lifelog retrieval systems. To demonstrate the effectiveness of the proposed pipeline, I integrated a lifelog QA model into MyScéal with modifications and developed a dedicated lifelog QA system known as MyEachtra. User studies were carried out to analyse the strengths and weak- nesses of MyEachtra. The results showed that MyEachtra effectively supports the user in answering lifelog questions and enhances overall user satisfaction. The findings of this research have the potential to establish a foundation for further exploration into the task of lifelog QA.
Metadata
Item Type:Thesis (PhD)
Date of Award:August 2024
Refereed:No
Supervisor(s):Gurrin, Cathal, Zhou, Liting and Conlan, Owen
Uncontrolled Keywords:Question Answering, Interactive Multimodal Retrieval
Subjects:Computer Science > Information retrieval
Computer Science > Information technology
Computer Science > Interactive computer systems
Computer Science > Machine learning
Computer Science > Multimedia systems
Computer Science > Lifelog
DCU Faculties and Centres:DCU Faculties and Schools > Faculty of Engineering and Computing > School of Computing
Use License:This item is licensed under a Creative Commons Attribution-NonCommercial-No Derivative Works 4.0 License. View License
Funders:SFI under grant agreement 13/RC/2106_P2 and the Centre for Research Training in Digitally-Enhanced Reality (d-real)
ID Code:29984
Deposited On:18 Nov 2024 14:25 by Ly Duyen Tran . Last Modified 18 Nov 2024 14:25
Documents

Full text available as:

[thumbnail of Final Thesis.pdf]
Preview
PDF - Requires a PDF viewer such as GSview, Xpdf or Adobe Acrobat Reader
Creative Commons: Attribution-Noncommercial-No Derivative Works 4.0
10MB
Downloads

Downloads

Downloads per month over past year

Archive Staff Only: edit this record