Crowdsourcing markets provide workers with a centralized place to find paid work. What
may not be obvious at first glance is that, in addition to the work they do for pay, crowd …

Modern information-seeking systems are becoming more interactive, mainly through asking
Clarifying Questions (CQs) to refine users' information needs. System-generated CQs may …

The use of clarifying questions (CQs) is a fairly new and useful technique to aid systems in
recognizing the intent, context, and preferences behind user queries. Yet, understanding the …

When collecting item ratings from human judges, it can be difficult to measure and enforce
data quality due to task subjectivity and lack of transparency into how judges make each …

We carried out a reproducibility study of InPars recipe for unsupervised training of neural
rankers. As a by-product of this study, we developed a simple-yet-effective modification of …

Paid micro-task crowdsourcing has gained in popularity partly due to the increasing need for
large-scale manually labelled datasets which are often used to train and evaluate Artificial …

Relevance is a key concept in information retrieval and widely used for the evaluation of
search systems using test collections. We present a comprehensive study of the effect of the …

In order to evaluate the effectiveness of Information Retrieval (IR) systems it is key to collect
relevance judgments from human assessors. Crowdsourcing has successfully been used as …

Misinformation has been rapidly spreading online. The common approach to dealing with it
is deploying expert fact-checkers who follow forensic processes to identify the veracity of …

Due to high annotation costs making the best use of existing human-created training data is
an important research direction. We, therefore, carry out a systematic evaluation of …