BERTHA: Video captioning evaluation via transfer-learned human assessment
Lebron, LuisORCID: 0000-0002-3230-3589, Graham, Yvette, McGuinness, KevinORCID: 0000-0003-1336-6477, Kouramas, Konstantinos and O'Connor, Noel E.ORCID: 0000-0002-4033-9135
(2022)
BERTHA: Video captioning evaluation via transfer-learned human assessment.
In: 13th Edition of its Language Resources and Evaluation Conference, 21--23 June 2022, Marseille, France.
Evaluating video captioning systems is a challenging task with multiple challenges to consider. Firstly, the fluency of the caption, multiple actions taking place within a single scene, and estimation of what a human user might consider important in a video. Most metrics aim to measure how similar the system generated captions are to a single or a set of human-generated captions. This paper presents a new method based on a deep learning model to evaluate systems. The model is based on BERT language model, shown to work well across a range of NLP tasks. The aim is for the model to learn to perform an evaluation similar to that of a human. To do so, we use a dataset that contains human evaluation of system-generated captions. The dataset consists of human judgments of the quality of captions produced by the system participating in past TRECVid video to text tasks. These annotations will be made publicly available.\footnotemark The new video captioning evaluation metric, BERTHA, obtains favourable results, outperforming commonly applied metrics in some setups.
Metadata
Item Type:
Conference or Workshop Item (Paper)
Event Type:
Conference
Refereed:
No
Uncontrolled Keywords:
Video captioning; NLP; deep learning; learned metric
This item is licensed under a Creative Commons Attribution-NonCommercial-Share Alike 3.0 License. View License
Funders:
Irish Research Council Enterprise Partnership Scheme together with United Technologies Research Center Ireland, Insight SFI Research Centre for Data Analytics supported by Science Foundation Ireland, SFI/12/RC/2289 P2, co-funded by the European Regional Development Fund.
ID Code:
27081
Deposited On:
20 Jun 2022 13:53 by
Luis Lebron Casas
. Last Modified 13 Jan 2023 12:08