Crowdspeech and voxdiy: Benchmark datasets for crowdsourced audio transcription
Domain-specific data is the crux of the successful transfer of machine learning systems from
benchmarks to real life. In simple problems such as image classification, crowdsourcing has …
benchmarks to real life. In simple problems such as image classification, crowdsourcing has …
A General Model for Aggregating Annotations Across Simple, Complex, and Multi-Object Annotation Tasks
Human annotations are vital to supervised learning, yet annotators often disagree on the
correct label, especially as annotation tasks increase in complexity. A common strategy to …
correct label, especially as annotation tasks increase in complexity. A common strategy to …
[HTML][HTML] An error consistency based approach to answer aggregation in open-ended crowdsourcing
Crowdsourcing plays a vital role in today's AI industry. However, existing crowdsourcing
research mainly focuses on those simple tasks that are often formulated as label …
research mainly focuses on those simple tasks that are often formulated as label …
Measuring annotator agreement generally across complex structured, multi-object, and free-text annotation tasks
When annotators label data, a key metric for quality assurance is inter-annotator agreement
(IAA): the extent to which annotators agree on their labels. Though many IAA measures exist …
(IAA): the extent to which annotators agree on their labels. Though many IAA measures exist …
A Comparative Study on Annotation Quality of Crowdsourcing and LLM via Label Aggregation
J Li - ICASSP 2024-2024 IEEE International Conference on …, 2024 - ieeexplore.ieee.org
Whether Large Language Models (LLMs) can outperform crowdsourcing on the data
annotation task is attracting interest recently. Some works verified this issue with the average …
annotation task is attracting interest recently. Some works verified this issue with the average …
Crowdsourced text sequence aggregation based on hybrid reliability and representation
J Li - Proceedings of the 43rd International ACM SIGIR …, 2020 - dl.acm.org
The crowd is cheaper and easier to access than the oracle to collect the ground truth data for
training and evaluating models. To ensure the quality of the crowdsourced data, people can …
training and evaluating models. To ensure the quality of the crowdsourced data, people can …
[PDF][PDF] A general-purpose crowdsourcing computational quality control toolkit for Python
D Ustalov, N Pavlichenko, V Losev… - The Ninth AAAI …, 2021 - humancomputation.com
Quality control is a crux of crowdsourcing. While most means for quality control are
organizational and imply worker selection, golden tasks, and post-acceptance …
organizational and imply worker selection, golden tasks, and post-acceptance …
Handwritten text recognition from crowdsourced annotations
In this paper, we explore different ways of training a model for handwritten text recognition
when multiple imperfect or noisy transcriptions are available. We consider various training …
when multiple imperfect or noisy transcriptions are available. We consider various training …
Learning from Crowds with Crowd-Kit
D Ustalov, N Pavlichenko, B Tseitlin - arXiv preprint arXiv:2109.08584, 2021 - arxiv.org
Quality control is a crux of crowdsourcing. While most means for quality control are
organizational and imply worker selection, golden tasks, and post-acceptance …
organizational and imply worker selection, golden tasks, and post-acceptance …
Crowdsourcing natural language data at scale: A hands-on tutorial
In this tutorial, we present a portion of unique industry experience in efficient natural
language data annotation via crowdsourcing shared by both leading researchers and …
language data annotation via crowdsourcing shared by both leading researchers and …