Skip to main content
DORAS
DCU Online Research Access Service
Login (DCU Staff Only)
An Exploration into the Benefits of the CLIP model for Lifelog Retrieval

Tran, Ly Duyen ORCID: 0000-0002-9597-1832, Alam, Naushad ORCID: 0000-0002-3144-5622, Vo, Linh Khanh, Diep, Nghiem Tuong, Nguyen, Binh ORCID: 0000-0001-5249-9702, Graham, Yvette ORCID: 0000-0001-6741-4855, Zhou, Liting ORCID: 0000-0002-7778-8743 and Gurrin, Cathal ORCID: 0000-0003-2903-3968 (2022) An Exploration into the Benefits of the CLIP model for Lifelog Retrieval. In: International Conference on Content-Based Multimedia Indexing, 14–16 Sept 2022, Graz, Austria. ISBN 978-1-4503-9720-9

Full text available as:

[img]
Preview
PDF - Requires a PDF viewer such as GSview, Xpdf or Adobe Acrobat Reader
2MB

Abstract

In this paper, we attempt to fine-tune the CLIP (Contrastive Language-Image Pre-Training) model on the Lifelog Question Answering dataset (LLQA) to investigate retrieval performance of the fine-tuned model over the zero-shot baseline model. We train the model adopting a weight space ensembling approach using a modified loss function to take into account the differences in our dataset (LLQA) when compared with the dataset the CLIP model was originally pretrained on. We further evaluate our fine-tuned model using visual as well as multimodal queries on multiple retrieval tasks, demonstrating improved performance over the zero-shot baseline model.

Item Type:Conference or Workshop Item (Paper)
Event Type:Conference
Refereed:Yes
Uncontrolled Keywords:lifelogging; image retrieval; pretrained models
Subjects:Computer Science > Algorithms
Computer Science > Information retrieval
Computer Science > Lifelog
DCU Faculties and Centres:DCU Faculties and Schools > Faculty of Engineering and Computing > School of Computing
Published in: International Conference on Content-Based Multimedia Indexing (CBMI 2022). . Association for Computing Machinery. ISBN 978-1-4503-9720-9
Publisher:Association for Computing Machinery
Official URL:https://doi.org/10.1145/3549555.3549593
Copyright Information:© 2022 The Authors (CC-BY 4.0)
Funders:Science Foundation Ireland grant numbers SFI/12/RC/2289, SFI/12/RC/2289-P2, SFI/13/RC/2106, 18/CRT/6223 and 18/CRT/6224
ID Code:27842
Deposited On:10 Oct 2022 09:09 by Ly Duyen Tran . Last Modified 10 Oct 2022 09:09

Downloads

Downloads per month over past year

Archive Staff Only: edit this record

  • Student Email
  • Staff Email
  • Student Apps
  • Staff Apps
  • Loop
  • Disclaimer
  • Privacy
  • Contact Us