Tran, Ly-Duyen ORCID: 0000-0002-9597-1832 (2024) Pushing the Boundaries of Lifelog Retrieval Systems with Question Answering Techniques. PhD thesis, Dublin City University.
Abstract
Lifelogging, referring to the continuous capturing of personal experiences using digital devices such as wearable cameras and smart sensors, could be a valuable memory enhance- ment and provide great insights into how an individual lives their life. This thesis focuses on the novel application of question answering (QA) within the context of lifelogging, aiming to develop an advanced interactive lifelog retrieval system that supports answering questions based on lifelog data. To achieve this objective, this research addresses several key components. First, a novel lifelog QA dataset, named LLQA, was created, consisting of over 15,000 multiple-choice and yes-no questions regarding data from lifelog segmenta- tions. I evaluated different existing QA models on their suitability and capability to an- swer lifelog questions and compared their accuracies to the human baseline of 85.46%. The evaluation results recognised the efficiency of leveraging large pre-trained video-language models, achieving an accuracy of 72.43%, as opposed to constructing custom-built LLQA models, which achieved 71.23%. Next, I designed and continually enhanced MyScéal, a state-of-the-art interactive lifelog retrieval system that supports the user to efficiently retrieve relevant data in response to search queries and lifelog questions. This effort cul- minated in MyScéal’s success as the winning system in three consecutive iterations of Lifelog Search Challenges, underscoring its strengths in supporting the user to quickly locate items of interest from a conventional multimodal lifelog. Finally, a novel lifelog QA pipeline was proposed to seamlessly integrate QA models into existing lifelog retrieval systems. To demonstrate the effectiveness of the proposed pipeline, I integrated a lifelog QA model into MyScéal with modifications and developed a dedicated lifelog QA system known as MyEachtra. User studies were carried out to analyse the strengths and weak- nesses of MyEachtra. The results showed that MyEachtra effectively supports the user in answering lifelog questions and enhances overall user satisfaction. The findings of this research have the potential to establish a foundation for further exploration into the task of lifelog QA.
Metadata
Item Type: | Thesis (PhD) |
---|---|
Date of Award: | August 2024 |
Refereed: | No |
Supervisor(s): | Gurrin, Cathal, Zhou, Liting and Conlan, Owen |
Uncontrolled Keywords: | Question Answering, Interactive Multimodal Retrieval |
Subjects: | Computer Science > Information retrieval Computer Science > Information technology Computer Science > Interactive computer systems Computer Science > Machine learning Computer Science > Multimedia systems Computer Science > Lifelog |
DCU Faculties and Centres: | DCU Faculties and Schools > Faculty of Engineering and Computing > School of Computing |
Use License: | This item is licensed under a Creative Commons Attribution-NonCommercial-No Derivative Works 4.0 License. View License |
Funders: | SFI under grant agreement 13/RC/2106_P2 and the Centre for Research Training in Digitally-Enhanced Reality (d-real) |
ID Code: | 29984 |
Deposited On: | 18 Nov 2024 14:25 by Ly Duyen Tran . Last Modified 18 Nov 2024 14:25 |
Documents
Full text available as:
Preview |
PDF
- Requires a PDF viewer such as GSview, Xpdf or Adobe Acrobat Reader
Creative Commons: Attribution-Noncommercial-No Derivative Works 4.0 10MB |
Downloads
Downloads
Downloads per month over past year
Archive Staff Only: edit this record