Machine Translation

The Machine Translation Journey

Machine Translation (MT) automatically translates text from a source language to a target language. It is one of the oldest and most studied NLP tasks, and its evolution mirrors the history of NLP itself.

Evolution of MT Approaches

Era	Approach	Example Systems
1950s–1990s	Rule-based (RBMT)	Systran, Logos — hand-crafted grammar rules
1990s–2010s	Statistical (SMT)	Moses — phrase tables + language models
2014–2017	Neural (RNN seq2seq)	Google NMT 2016 — encoder-decoder + attention
2017–now	Transformer	MarianMT, mBART, M2M-100, NLLB-200

The Noisy Channel Model (SMT)

Statistical MT frames translation as finding the most probable target sentence $mathbf{e}$ given source $mathbf{f}$:

$hat{e} = \arg\max_e P(e|f) = \arg\max_e \underbrace{P(f|e)}_{\text{translation model}} \cdot \underbrace{P(e)}_{\text{language model}}$

• Translation model $P(f|e)$: learned from parallel corpora (phrase-table)
• Language model $P(e)$: ensures fluent output
• Decoder: beam search over exponentially many hypotheses

Limitations of SMT: phrase tables are brittle; no long-range context; separate components poorly integrated; requires language-pair specific pipelines.

Neural Machine Translation (NMT)

RNN Encoder-Decoder

The basic seq2seq architecture uses LSTMs:

Source: "Die Katze sitzt auf der Matte"
  ↓  Encoder (bidirectional LSTM)
  h1   h2   h3   h4   h5   h6
  ↓  Context vector c = h6 (last hidden state)
  ↓  Decoder (LSTM, conditioned on c)
 "The" "cat" "sits" "on" "the" "mat" </s>

Bottleneck problem: the entire source sentence must be compressed into a single fixed-size context vector. This fails for long sentences.

Attention Mechanism (Bahdanau, 2015)

Attention allows the decoder to selectively focus on different parts of the source at each generation step:

$\text{score}(s_t, h_j) = \mathbf{v}_a^T \tanh(W_a s_t + U_a h_j)$

$\alpha_{tj} = \frac{\exp(\text{score}(s_t, h_j))}{\sum_{j'} \exp(\text{score}(s_t, h_{j'}))}$

$c_t = \sum_j \alpha_{tj} h_j$

• $s_t$: decoder hidden state at step $t$
• $h_j$: encoder hidden state for source token $j$
• $\alpha_{tj}$: attention weight (soft alignment) between target position $t$ and source position $j$
• $c_t$: context vector — weighted sum of encoder states

Attention weights can be visualized as an alignment matrix, revealing which source words the model focuses on when generating each target word.

The Transformer for MT

The Transformer (Vaswani et al., 2017, "Attention Is All You Need") replaces recurrence with multi-head self-attention:

$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right) V$

Why Transformers dominate MT:

Property	RNN Seq2Seq	Transformer
Parallelism	Sequential (hard to GPU-parallelize)	Fully parallel
Long-range context	Degrades with distance	Direct $O(1)$ connections
Training speed	Slow (sequential)	Fast (parallel)
Performance	Good	SOTA

Cross-attention in the decoder attends to encoder outputs — the Transformer version of Bahdanau attention. Self-attention in the encoder allows each source token to attend to all others, building rich contextual representations.

# ─── 1. MarianMT — Fast Language-Pair Specific MT ────────────
from transformers import MarianMTModel, MarianTokenizer

def translate(text, src_lang="en", tgt_lang="fr"):
    model_name = f"Helsinki-NLP/opus-mt-{src_lang}-{tgt_lang}"
    tokenizer = MarianTokenizer.from_pretrained(model_name)
    model = MarianMTModel.from_pretrained(model_name)

    inputs = tokenizer(text, return_tensors="pt", padding=True, truncation=True)
    translated = model.generate(**inputs, num_beams=4, max_length=512)
    return [tokenizer.decode(t, skip_special_tokens=True) for t in translated]

sentences = [
    "Natural language processing enables machines to understand human language.",
    "The quick brown fox jumps over the lazy dog.",
    "Machine translation has improved dramatically with neural networks.",
]
print("English → French:")
for src, tgt in zip(sentences, translate(sentences)):
    print(f"  EN: {src}")
    print(f"  FR: {tgt}\n")

# ─── 2. M2M-100 — Multilingual (100 languages, any pair) ──────
from transformers import M2M100ForConditionalGeneration, M2M100Tokenizer

m2m_tokenizer = M2M100Tokenizer.from_pretrained("facebook/m2m100_418M")
m2m_model = M2M100ForConditionalGeneration.from_pretrained("facebook/m2m100_418M")

def m2m_translate(text, src_lang, tgt_lang):
    m2m_tokenizer.src_lang = src_lang
    encoded = m2m_tokenizer(text, return_tensors="pt")
    tgt_lang_id = m2m_tokenizer.get_lang_id(tgt_lang)
    generated = m2m_model.generate(
        **encoded,
        forced_bos_token_id=tgt_lang_id,
        num_beams=5,
        max_length=200,
    )
    return m2m_tokenizer.decode(generated[0], skip_special_tokens=True)

# Japanese → Spanish (no English pivot needed!)
result = m2m_translate("自然言語処理は人工知能の重要な分野です。", "ja", "es")
print(f"JA→ES: {result}")

# ─── 3. NLLB-200 — No Language Left Behind ───────────────────
# Meta's model supporting 200 languages including low-resource ones
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

nllb_tokenizer = AutoTokenizer.from_pretrained("facebook/nllb-200-distilled-600M")
nllb_model = AutoModelForSeq2SeqLM.from_pretrained("facebook/nllb-200-distilled-600M")

def nllb_translate(text, src_bcp47, tgt_bcp47):
    """BCP-47 codes: 'eng_Latn', 'fra_Latn', 'swh_Latn' (Swahili), etc."""
    nllb_tokenizer.src_lang = src_bcp47
    inputs = nllb_tokenizer(text, return_tensors="pt")
    tgt_id = nllb_tokenizer.lang_code_to_id[tgt_bcp47]
    output = nllb_model.generate(
        **inputs,
        forced_bos_token_id=tgt_id,
        num_beams=4,
        max_length=200,
    )
    return nllb_tokenizer.decode(output[0], skip_special_tokens=True)

en_text = "Climate change is one of the greatest challenges of our time."
print(f"EN: {en_text}")
print(f"FR: {nllb_translate(en_text, 'eng_Latn', 'fra_Latn')}")
print(f"SW: {nllb_translate(en_text, 'eng_Latn', 'swh_Latn')}")  # Swahili

# ─── 4. BLEU Score Evaluation ────────────────────────────────
from sacrebleu.metrics import BLEU, CHRF, TER
import sacrebleu

references = [
    ["The cat sat on the mat.", "A cat was sitting on the mat."],
    ["Natural language processing is a field of artificial intelligence."],
]
hypotheses = [
    "The cat is sitting on the mat.",
    "Natural language processing is an area of AI.",
]

bleu = BLEU()
chrf = CHRF()

print("\nMT Evaluation Metrics:")
for i, (hyp, refs) in enumerate(zip(hypotheses, references)):
    bleu_score = sacrebleu.corpus_bleu([hyp], [refs])
    chrf_score = sacrebleu.corpus_chrf([hyp], [refs])
    print(f"\n  Hypothesis {i+1}: {hyp}")
    print(f"  Reference(s): {refs}")
    print(f"  BLEU: {bleu_score.score:.2f}, chrF: {chrf_score.score:.2f}")

# ─── 5. Attention Visualization ──────────────────────────────
import torch
import matplotlib.pyplot as plt

model_name = "Helsinki-NLP/opus-mt-en-de"
tok = MarianTokenizer.from_pretrained(model_name)
mdl = MarianMTModel.from_pretrained(model_name)

text = "The bank can guarantee deposits will eventually cover future tuition costs."
inputs = tok(text, return_tensors="pt")
with torch.no_grad():
    outputs = mdl.generate(
        **inputs, return_dict_in_generate=True,
        output_attentions=True, num_beams=1
    )
print(f"\nTranslation (EN→DE): {tok.decode(outputs.sequences[0], skip_special_tokens=True)}")
# Cross-attention weights available in outputs.cross_attentions

MT Evaluation Metrics

BLEU (Bilingual Evaluation Understudy)

BLEU measures n-gram precision between hypothesis and reference translations:

$\text{BLEU} = \text{BP} \cdot \exp\left(\sum_{n=1}^{N} w_n \log p_n\right)$

• $p_n$: modified n-gram precision (1-gram to 4-gram), with clipping
• $\text{BP}$: brevity penalty to discourage short translations
• $w_n = 1/N$: uniform weights (typically $N=4$)
• Score range: 0–100 (higher = better)

BLEU interpretation guide:

Score	Quality
< 10	Near unusable
10–19	Poor, gist intelligible
20–29	Understandable with effort
30–40	Good, fluent in parts
40–50	High quality
50–60	Very high quality / near human
> 60	Human or better (rare)

Limitations of BLEU: sensitive to tokenization; doesn't measure fluency; fails for morphologically rich languages; misses synonyms and paraphrases.

chrF and COMET

• chrF (character n-gram F-score): operates on character n-grams, better for morphologically rich languages and low-resource settings
• COMET (Crosslingual Optimized Metric for Evaluation of Translation): a learned metric trained on human quality judgments; correlates much better with human evaluations than BLEU

Multilingual and Low-Resource MT

Challenges in low-resource MT:

• Few parallel sentences (<100k pairs)
• Solutions: back-translation, multilingual pre-training (mBART, M2M-100), transfer from related languages

NLLB-200 (Meta, 2022): covers 200 languages including 55 African languages. Uses language family groupings and massive multilingual pre-training to achieve acceptable quality even for extremely low-resource pairs.

Summary

• Rule-based MT used hand-crafted grammar rules — high precision for known patterns, poor generalization
• Statistical MT (noisy channel model): translation + language model, learned from parallel corpora via phrase tables
• Neural MT (seq2seq + attention): LSTM encoder-decoder with soft alignment; solved the bottleneck problem
• Transformer MT: parallel self-attention replaces sequential RNNs; faster training, better long-range context, SOTA quality
• Multilingual MT: M2M-100 and NLLB-200 enable direct any-to-any translation across 100–200 languages
• BLEU is the standard metric but has known limitations; COMET better correlates with human judgment

Next: Question Answering & Summarization — making models answer questions and condense information.