Machine learning classifiers for human-facing tasks such as comment toxicity and
misinformation often score highly on metrics such as ROC AUC but are received poorly in …

The labels used to train machine learning (ML) models are of paramount importance.
Typically for ML classification tasks, datasets contain hard labels, yet learning using soft …

When crowdsourcing the creation of machine learning datasets, statistical distributions that
capture diverse answers can represent ambiguous data better than a single best answer …

Emerging crowdsensing paradigm enables a large number of sensing applications, where
much attention is drawn to the fundamental problems of data collection and truth inference …

Individual human decision-makers may benefit from different forms of support to improve
decision outcomes. However, a key question is which form of support will lead to accurate …

In this work we explore confidence elicitation methods for crowdsourcing" soft" labels, eg,
probability estimates, to reduce the annotation costs for domains with ambiguous data …

Crowdsourcing is a widely adopted way for various labor-intensive tasks. One of the core
problems in crowdsourcing systems is how to assign tasks to most suitable workers for better …

Research on email anomaly detection has typically relied on specially prepared datasets
that may not adequately reflect the type of data that occurs in industry settings. In our …

The quality of crowdsourced data is often highly variable. For this reason, it is common to
collect redundant data and use statistical methods to aggregate it. Empirical studies show …

Developing machine learning models worthy of decision-maker trust is crucial to using
models in practice. Algorithmic transparency tools, such as explainability and uncertainty …