Automatic MT Evaluation

Lecture 4: Automatic MT Evaluation

Lecture video:	web TODO Youtube

{{#ev:youtube|https://www.youtube.com/watch?v=Bj_Hxi91GUM&index=5&list=PLpiLOsNLsfmbeH-b865BwfH15W0sat02V%7C800%7Ccenter}}

Reference Translations

The following picture illustrates the issue of reference translations:

Out of all possible sequences of words in the given language, only some are grammatically correct sentences ($G$). An overlapping set is formed by understandable translations of the source sentence (note that these are not necessarily grammatical). Possible reference translations can then be viewed as a subset of the interse

Despite this fact, when we train or evaluate translation systems, we often rely on just a single reference translation.

Translation Evaluation Campaigns

There are several academic workshops where the quality of various translation systems is compared. Such "competitions" require manual evaluation. Their methodology evolves to make the results as fair and statistically sound as possible. The most prominent ones include:

Workshop on Statistical Machine Translation (WMT)

International Workshop on Spoken Language Translation (IWSLT)