Knowledge learning with crowdsourcing: A brief review and systematic perspective
J Zhang - IEEE/CAA Journal of Automatica Sinica, 2022 - ieeexplore.ieee.org
Big data have the characteristics of enormous volume, high velocity, diversity, value-sparsity,
and uncertainty, which lead the knowledge learning from them full of challenges. With the …
and uncertainty, which lead the knowledge learning from them full of challenges. With the …
If in a Crowdsourced Data Annotation Pipeline, a GPT-4
Recent studies indicated GPT-4 outperforms online crowd workers in data labeling
accuracy, notably workers from Amazon Mechanical Turk (MTurk). However, these studies …
accuracy, notably workers from Amazon Mechanical Turk (MTurk). However, these studies …
Truth inference at scale: A bayesian model for adjudicating highly redundant crowd annotations
Crowd-sourcing is a cheap and popular means of creating training and evaluation datasets
for machine learning, however it poses the problem of'truth inference', as individual workers …
for machine learning, however it poses the problem of'truth inference', as individual workers …
Graph contrastive learning for truth inference
Crowdsourcing has become a popular paradigm for collecting large-scale labeled datasets
by leveraging numerous annotators. However, these annotators often provide noisy labels …
by leveraging numerous annotators. However, these annotators often provide noisy labels …
A little truth injection but a big reward: Label aggregation with graph neural networks
Various correlations hidden in crowdsourcing annotation tasks bring opportunities to further
improve the accuracy of label aggregation. However, these relationships are usually …
improve the accuracy of label aggregation. However, these relationships are usually …
Exploiting predicted answer in label aggregation to make better use of the crowd wisdom
Nowadays, crowdsourcing is a widespread and effective method to gather the crowd
wisdom. At the same time, label aggregation is used to aggregate the noisy and biased data …
wisdom. At the same time, label aggregation is used to aggregate the noisy and biased data …
On the supervision of peer assessment tasks: An efficient instructor guidance technique
J Hernández-González… - IEEE Transactions on …, 2023 - ieeexplore.ieee.org
In peer assessment, students assess a task done by their peers, provide feedback and
usually a grade. The extent to which these peer grades can be used to formally grade the …
usually a grade. The extent to which these peer grades can be used to formally grade the …
Exploiting heterogeneous graph neural networks with latent worker/task correlation information for label aggregation in crowdsourcing
Crowdsourcing has attracted much attention for its convenience to collect labels from non-
expert workers instead of experts. However, due to the high level of noise from the non …
expert workers instead of experts. However, due to the high level of noise from the non …
Crowdsourcing truth inference via reliability-driven multi-view graph embedding
Crowdsourcing truth inference aims to assign a correct answer to each task from candidate
answers that are provided by crowdsourced workers. A common approach is to generate …
answers that are provided by crowdsourced workers. A common approach is to generate …
Towards Long-term Annotators: A Supervised Label Aggregation Baseline
Relying on crowdsourced workers, data crowdsourcing platforms are able to efficiently
provide vast amounts of labeled data. Due to the variability in the annotation quality of crowd …
provide vast amounts of labeled data. Due to the variability in the annotation quality of crowd …