Qa dataset explosion: A taxonomy of nlp resources for question answering and reading comprehension
Alongside huge volumes of research on deep learning models in NLP in the recent years,
there has been much work on benchmark datasets needed to track modeling progress …
there has been much work on benchmark datasets needed to track modeling progress …
A survey of evaluation metrics used for NLG systems
In the last few years, a large number of automatic evaluation metrics have been proposed for
evaluating Natural Language Generation (NLG) systems. The rapid development and …
evaluating Natural Language Generation (NLG) systems. The rapid development and …
Benchmarking foundation models with language-model-as-an-examiner
Numerous benchmarks have been established to assess the performance of foundation
models on open-ended question answering, which serves as a comprehensive test of a …
models on open-ended question answering, which serves as a comprehensive test of a …
Evaluating open-domain question answering in the era of large language models
Lexical matching remains the de facto evaluation method for open-domain question
answering (QA). Unfortunately, lexical matching fails completely when a plausible candidate …
answering (QA). Unfortunately, lexical matching fails completely when a plausible candidate …
[HTML][HTML] Human evaluation of automatically generated text: Current trends and best practice guidelines
Currently, there is little agreement as to how Natural Language Generation (NLG) systems
should be evaluated, with a particularly high degree of variation in the way that human …
should be evaluated, with a particularly high degree of variation in the way that human …
Selective question answering under domain shift
To avoid giving wrong answers, question answering (QA) models need to know when to
abstain from answering. Moreover, users often ask questions that diverge from the model's …
abstain from answering. Moreover, users often ask questions that diverge from the model's …
Find or classify? dual strategy for slot-value predictions on multi-domain dialog state tracking
Dialog state tracking (DST) is a core component in task-oriented dialog systems. Existing
approaches for DST mainly fall into one of two categories, namely, ontology-based and …
approaches for DST mainly fall into one of two categories, namely, ontology-based and …
Human evaluation of conversations is an open problem: comparing the sensitivity of various methods for evaluating dialogue agents
At the heart of improving conversational AI is the open problem of how to evaluate
conversations. Issues with automatic metrics are well known (Liu et al., 2016, arXiv …
conversations. Issues with automatic metrics are well known (Liu et al., 2016, arXiv …
Tomayto, tomahto. beyond token-level answer equivalence for question answering evaluation
The predictions of question answering (QA) systems are typically evaluated against
manually annotated finite sets of one or more answers. This leads to a coverage limitation …
manually annotated finite sets of one or more answers. This leads to a coverage limitation …
A survey on machine reading comprehension systems
Machine Reading Comprehension (MRC) is a challenging task and hot topic in Natural
Language Processing. The goal of this field is to develop systems for answering the …
Language Processing. The goal of this field is to develop systems for answering the …