The TRECVID 2007 BBC rushes summarization evaluation pilot
Over, Paul and Smeaton, Alan F. and Kelly, Philip (2007) The TRECVID 2007 BBC rushes summarization evaluation pilot. In: TVS 2007 - TRECVID BBC Rushes Summarization Workshop, 24-29 September 2007, Augsburg, Germany.
Full text available as:
This paper provides an overview of a pilot evaluation of
video summaries using rushes from several BBC dramatic series. It was carried out under the auspices of TRECVID.
Twenty-two research teams submitted video summaries of
up to 4% duration, of 42 individual rushes video files aimed
at compressing out redundant and insignificant material.
The output of two baseline systems built on straightforward
content reduction techniques was contributed by Carnegie
Mellon University as a control. Procedures for developing
ground truth lists of important segments from each video
were developed at Dublin City University and applied to
the BBC video. At NIST each summary was judged by
three humans with respect to how much of the ground truth
was included, how easy the summary was to understand,
and how much repeated material the summary contained.
Additional objective measures included: how long it took
the system to create the summary, how long it took the assessor to judge it against the ground truth, and what the
summary's duration was. Assessor agreement on finding desired segments averaged 78% and results indicate that while it is difficult to exceed the performance of baselines, a few systems did.
Archive Staff Only: edit this record