Towards explainable evaluation metrics for machine translation
Unlike classical lexical overlap metrics such as BLEU, most current evaluation metrics for
machine translation (for example, COMET or BERTScore) are based on black-box large …
machine translation (for example, COMET or BERTScore) are based on black-box large …
Machine translation meta evaluation through translation accuracy challenge sets
Recent machine translation (MT) metrics calibrate their effectiveness by correlating with
human judgment. However, these results are often obtained by averaging predictions across …
human judgment. However, these results are often obtained by averaging predictions across …
Metric score landscape challenge (MSLC23): Understanding metrics' performance on a wider landscape of translation quality
Abstract The Metric Score Landscape Challenge (MSLC23) dataset aims to gain insight into
metric scores on a broader/wider landscape of machine translation (MT) quality. It provides a …
metric scores on a broader/wider landscape of machine translation (MT) quality. It provides a …
ACES: Translation accuracy challenge sets at WMT 2023
We benchmark the performance of segmentlevel metrics submitted to WMT 2023 using the
ACES Challenge Set (Amrhein et al., 2022). The challenge set consists of 36K examples …
ACES Challenge Set (Amrhein et al., 2022). The challenge set consists of 36K examples …
BLEU Meets COMET: Combining Lexical and Neural Metrics Towards Robust Machine Translation Evaluation
Although neural-based machine translation evaluation metrics, such as COMET or BLEURT,
have achieved strong correlations with human judgements, they are sometimes unreliable in …
have achieved strong correlations with human judgements, they are sometimes unreliable in …
Multifaceted Challenge Set for Evaluating Machine Translation Performance
Abstract Machine Translation Evaluation is critical to Machine Translation research, as the
evaluation results reflect the effectiveness of training strategies. As a result, a fair and …
evaluation results reflect the effectiveness of training strategies. As a result, a fair and …
Robustness Tests for Automatic Machine Translation Metrics with Adversarial Attacks
Y Huang, T Baldwin - arXiv preprint arXiv:2311.00508, 2023 - arxiv.org
We investigate MT evaluation metric performance on adversarially-synthesized texts, to
shed light on metric robustness. We experiment with word-and character-level attacks on …
shed light on metric robustness. We experiment with word-and character-level attacks on …
Pulling Out All The Full Stops: Punctuation Sensitivity in Neural Machine Translation and Evaluation
P Jwalapuram - Findings of the Association for Computational …, 2023 - aclanthology.org
Much of the work testing machine translation systems for robustness and sensitivity has
been adversarial or tended towards testing noisy input such as spelling errors, or non …
been adversarial or tended towards testing noisy input such as spelling errors, or non …
Segment-level evaluation of machine translation metrics
N Moghe - 2024 - era.ed.ac.uk
Most metrics evaluating Machine Translation (MT) claim their effectiveness by demonstrating
their ability to distinguish the quality of different MT systems over a large corpus (system …
their ability to distinguish the quality of different MT systems over a large corpus (system …
[PDF][PDF] Evaluation of Pre-trained Metrics and ChatGPT as Document-level Machine Translation Metrics
N Bleiker - cl.uzh.ch
Automatic evaluation metrics play an important role in the development and optimization of
machine translation (MT) systems as they are the main method used for evaluating and …
machine translation (MT) systems as they are the main method used for evaluating and …