Automatic MT Evaluation: Difference between revisions

From MT Talks
Jump to navigation Jump to search
No edit summary
No edit summary
Line 14: Line 14:
[[File:references.png|650px]]
[[File:references.png|650px]]


Out of all possible sequences of words in the given language, only some are ''grammatically correct sentences'' (<math>G</math>). An overlapping set is formed by ''understandable translations''  (<math>T</math>) of the source sentence (note that these are not necessarily grammatical). Possible ''reference translations'' can then be viewed as a subset of <math>G \cup T</math>. Only some of these can be reached by the MT system. Typically, we only have several reference translations at our disposal; often just a single reference.
Out of all possible sequences of words in the given language, only some are ''grammatically correct sentences'' (<math>G</math>). An overlapping set is formed by ''understandable translations''  (<math>T</math>) of the source sentence (note that these are not necessarily grammatical). Possible ''reference translations'' can then be viewed as a subset of <math>G \cup T</math>. Only some of these can be reached by the MT system. Typically, we only have several reference translations at our disposal; often we have just a single reference.


== Contents ==
== PER ==


=== test ===
Position-independent error rate<ref name="per">C. Tillmann, S. Vogel, H. Ney, A. Zubiaga, H. Sawaf. ''[https://www-i6.informatik.rwth-aachen.de/publications/download/203/TillmannC.VogelS.NeyH.SawafH.ZubiagaA.--AcceleratedDP-basedSearchforStatisticalTranslation--1997.pdf Accelerated DP Based Search for Statistical Translation]''</ref> (PER)
 
== BLEU ==


== References ==
== References ==


<references />
<references />

Revision as of 18:54, 9 February 2015

Lecture 4: Automatic MT Evaluation
Lecture video: web TODO
Youtube

{{#ev:youtube|https://www.youtube.com/watch?v=Bj_Hxi91GUM&index=5&list=PLpiLOsNLsfmbeH-b865BwfH15W0sat02V%7C800%7Ccenter}}

Reference Translations

The following picture[1] illustrates the issue of reference translations:

Out of all possible sequences of words in the given language, only some are grammatically correct sentences (). An overlapping set is formed by understandable translations () of the source sentence (note that these are not necessarily grammatical). Possible reference translations can then be viewed as a subset of . Only some of these can be reached by the MT system. Typically, we only have several reference translations at our disposal; often we have just a single reference.

PER

Position-independent error rate[2] (PER)

BLEU

References

  1. Ondřej Bojar, Matouš Macháček, Aleš Tamchyna, Daniel Zeman. Scratching the Surface of Possible Translations
  2. C. Tillmann, S. Vogel, H. Ney, A. Zubiaga, H. Sawaf. Accelerated DP Based Search for Statistical Translation