Scoring and Optimization

Lecture 13: Scoring and Optimization

Lecture video:	web TODO Youtube

{{#ev:youtube|https://www.youtube.com/watch?v=oxhc0Nv_ySw&index=11&list=PLpiLOsNLsfmbeH-b865BwfH15W0sat02V%7C800%7Ccenter}}

Features of MT Models

Phrase Translation Probabilities

Phrase translation probabilities are calculated from occurrences of phrase pairs extracted from the parallel training data. Usually, MT systems work with the following two conditional probabilities:

• $P(\mathbf {e} |\mathbf {f} )$
• $P(\mathbf {f} |\mathbf {e} )$

These probabilities are estimated by simply counting how many times (for the first formula) we saw $\mathbf {e}$ aligned to $\mathbf {f}$ and how many times we saw $\mathbf {f}$ in total. For example, based on the following excerpt from (sorted) extracted phrase pairs, we estimate that $P({\text{naznačena v programu}}|{\text{estimated in the programme}})=3/9$.

estimated in the programme ||| naznačena v programu
estimated in the programme ||| naznačena v programu
estimated in the programme ||| naznačena v programu
estimated in the programme ||| odhadován v programu
estimated in the programme ||| odhadovány v programu
estimated in the programme ||| odhadovány v programu 
estimated in the programme ||| předpokládal program
estimated in the programme ||| v programu uvedeným
estimated in the programme ||| v programu uvedeným

Lexical Weights

Lexical weights are a method for smoothing the phrase table. Infrequent phrases have unreliable probability estimates; for instance many long phrases occur together only once in the corpus, resulting in $P(\mathbf {e} |\mathbf {f} )=P(\mathbf {f} |\mathbf {e} )=1$. Several methods exist for computing lexical weights. The most common one is based on word alignment inside the phrase. The probability of each foreign word $f_{j}$ is estimated as the average of lexical translation probabilities $w(f_{j},e_{i})$ over the English words aligned to it. Thus for the phrase $(\mathbf {e} ,\mathbf {f} )$ with the set of alignment points $a$, the lexical weight is:

${\text{lex}}(\mathbf {f} |\mathbf {e} ,a)=\prod _{j=1}^{l_{f}}{\frac {1}{|{i|(i,j)\in a}|}}\sum _{\forall (i,j)\in a}w(f_{j},e_{i})$

Language Model

The task of language modeling in machine translation is to estimate how likely a sequence of words $\mathbf {w} =(w_{1},\ldots ,w_{l})$ is in the target language.

When translating, the decoder generates translation hypotheses which are probable according to the translation model (i.e. the phrase table). The language model then scores these hypotheses according to how probable (common, fluent) they are in the target language. The final translation is then something like a compromise -- the sentence that is both fluent and a good translation of the input.

Similarly to the translation model, sequence probabilities are learned from data using maximum likelihood estimation. For language modeling, only monolingual data are needed (a resource available in much larger amounts than parallel texts).

Naturally, the prediction of the whole sequence $\mathbf {e}$ has to be decomposed, so that it can be reliably estimated. The most common approach are n-gram language models which build upon the Markov assumption: a word depends only on a limited, fixed number of preceding words. The decomposition is done as follows:

${\begin{aligned}P(\mathbf {w} )&=P(w_{1})P(w_{2}|w_{1})P(w_{3}|w_{1},w_{2})\ldots P(w_{l}|w_{1},\ldots ,w_{l-1})\\&\approx P(w_{1})P(w_{2}|w_{1})\ldots P(w_{l}|w_{l-n},\ldots ,w_{l-1})\end{aligned}}$

The first equality follows from the chain rule and the second from n-th order Markov assumption. Each word is then modeled by at most n preceding words and the probability of the whole sequence is the product of probabilities of individual words. Smoothing is further used to supply probability estimates to unseen n-grams.

A great introduction to language modeling is the video lecture by Jason Eisner. LMs are covered in more depth in the Stanford NLP lectures on Coursera; videos from the Coursera course can be found on YouTube.

Word and Phrase Penalty

For each word and for each phrase produced, the decoder pays a constant cost. Tweaking the word penalty can lead to either very short or very long output sentences (the "penalty" can also be negative -- a reward). Changes to the phrase penalty can lead to outputs consisting of word-by-word translations (small or negative phrase penalty -- use as many phrases as possible) or on the other hand, to outputs consisting of very long phrases (as is usually desirable).

Optimization of Feature Weights

Note that there have even been shared tasks in model optimization. One, by invitation only, in 2011 and one in 2015: WMT15 Tuning Task.