To prevent the costly and inefficient use of resources on low-quality annotations, we want a
method for creating a pool of dependable annotators who can effectively complete difficult …

Automatic summarization has consistently attracted attention, due to its versatility and wide
application in various downstream tasks. Despite its popularity, we find that annotation …

Human evaluation is the foundation upon which the evaluation of both summarization
systems and automatic metrics rests. However, existing human evaluation studies for …

While human evaluation remains best practice for accurately judging the faithfulness of
automatically-generated summaries, few solutions exist to address the increased difficulty …

Automatic text summarization aims to produce a brief but crucial summary for the input
documents. Both extractive and abstractive methods have witnessed great success in …

There has been substantial progress in summarization research enabled by the availability
of novel, often large-scale, datasets and recent advances on neural network-based …

State-of-the-art summarization systems are trained and evaluated on massive datasets
scraped from the web. Despite their prevalence, we know very little about the underlying …

Existing benchmarks for summarization quality evaluation often lack diverse input scenarios,
focus on narrowly defined dimensions (eg, faithfulness), and struggle with subjective and …

Existing summarization systems mostly generate summaries purely relying on the content of
the source document. However, even for humans, we usually need some references or …

Expert human annotation for summarization is definitely an expensive task, and can not be
done on huge scales. But with this work, we show that even with a crowd sourced summary …