Automatic MT Evaluation

Lecture 4: Automatic MT Evaluation

Lecture video:	web TODO Youtube

{{#ev:youtube|https://www.youtube.com/watch?v=Bj_Hxi91GUM&index=5&list=PLpiLOsNLsfmbeH-b865BwfH15W0sat02V%7C800%7Ccenter}}

Reference Translations

The following picture illustrates the issue of reference translations:

Out of all possible sequences of words in the given language, only some are grammatically correct sentences ($G$). An overlapping set is formed by understandable translations ($T$) of the source sentence (note that these are not necessarily grammatical). Possible reference translations can then be viewed as a subset of $G\cap T$. Only some of these can be reached by the MT system. Typically, we only have several reference translations at our disposal; often we have just a single reference.

PER

Position-independent error rate (PER) is a simple measure which counts the number of words which are identical in the MT output and the reference translation and divides