Human variation in labeling is often considered noise. Annotation projects for machine
learning (ML) aim at minimizing human label variation, with the assumption to maximize …

Abstract Many tasks in Natural Language Processing (NLP) and Computer Vision (CV) offer
evidence that humans disagree, from objective tasks such as part-of-speech tagging to more …

The use of misogynistic and sexist language has increased in recent years in social media,
and is increasing in the Arabic world in reaction to reforms attempting to remove restrictions …

Calibration is a popular framework to evaluate whether a classifier knows when it does not
know-ie, its predictive probabilities are a good indication of how likely a prediction is to be …

Supervised machine learning approaches often rely on a" ground truth" label. However,
obtaining one label through majority voting ignores the important subjectivity information in …

Crowdsourced data are often rife with disagreement, either because of genuine item
ambiguity, overlapping labels, subjectivity, or annotator error. Hence, a variety of methods …

We present Superlim, a multi-task NLP benchmark and analysis platform for evaluating
Swedish language models, a counterpart to the English-language (Super) GLUE suite. We …

Human annotations are vital to supervised learning, yet annotators often disagree on the
correct label, especially as annotation tasks increase in complexity. A common strategy to …

Although several datasets annotated for anaphoric reference/coreference exist, even the
largest such datasets have limitations in terms of size, range of domains, coverage of …

When annotators label data, a key metric for quality assurance is inter-annotator agreement
(IAA): the extent to which annotators agree on their labels. Though many IAA measures exist …