The DCU dependency-based metric in WMT-MetricsMATR 2010

Abstract

We describe DCU’s LFG dependencybased metric submitted to the shared evaluation task of WMT-MetricsMATR 2010. The metric is built on the LFG F-structurebased approach presented in (Owczarzak et al., 2007). We explore the following improvements on the original metric: 1) we replace the in-house LFG parser with an open source dependency parser that directly parses strings into LFG dependencies; 2) we add a stemming module and unigram paraphrases to strengthen the aligner; 3) we introduce a chunk penalty following the practice of METEOR to reward continuous matches; and 4) we introduce and tune parameters to maximize the correlation with human judgement. Experiments show that these enhancements improve the dependency-based metric's correlation with human judgement.