AI models are increasingly applied in high-stakes domains like health and conservation.
Data quality carries an elevated significance in high-stakes AI due to its heightened …

Objective: To provide a human–Artificial Intelligence (AI) interaction review for Machine
Learning (ML) applications to inform how to best combine both human domain expertise and …

Whose labels should a machine learning (ML) algorithm learn to emulate? For ML tasks
ranging from online comment toxicity to misinformation detection to medical diagnosis …

People who design, use, and are affected by autonomous artificially intelligent agents want
to be able to trust such agents—that is, to know that these agents will perform correctly, to …

Data collection is a major bottleneck in machine learning and an active research topic in
multiple communities. There are largely two reasons data collection has recently become a …

The potential for machine learning (ML) systems to amplify social inequities and unfairness
is receiving increasing popular and academic attention. A surge of recent work has focused …

Machine learning classifiers for human-facing tasks such as comment toxicity and
misinformation often score highly on metrics such as ROC AUC but are received poorly in …

Recent technological advances, especially in the field of machine learning, provide
astonishing progress on the road towards artificial general intelligence. However, tasks in …

In recent years, progress in NLU has been driven by benchmarks. These benchmarks are
typically collected by crowdsourcing, where annotators write examples based on annotation …

Biomedical image analysis algorithm validation depends on high-quality annotation of
reference datasets, for which labelling instructions are key. Despite their importance, their …