TY - GEN
T1 - The face of quality in crowdsourcing relevance labels
T2 - 21st ACM International Conference on Information and Knowledge Management, CIKM 2012
AU - Kazai, Gabriella
AU - Kamps, Jaap
AU - Milic-Frayling, Natasa
PY - 2012
Y1 - 2012
N2 - Information retrieval systems require human contributed relevance labels for their training and evaluation. Increasingly such labels are collected under the anonymous, uncontrolled conditions of crowdsourcing, leading to varied output quality. While a range of quality assurance and control techniques have now been developed to reduce noise during or after task completion, little is known about the workers themselves and possible relationships between workers' characteristics and the quality of their work. In this paper, we ask how do the relatively well or poorly-performing crowds, working under specific task conditions, actually look like in terms of worker characteristics, such as demographics or personality traits. Our findings show that the face of a crowd is in fact indicative of the quality of their work.
AB - Information retrieval systems require human contributed relevance labels for their training and evaluation. Increasingly such labels are collected under the anonymous, uncontrolled conditions of crowdsourcing, leading to varied output quality. While a range of quality assurance and control techniques have now been developed to reduce noise during or after task completion, little is known about the workers themselves and possible relationships between workers' characteristics and the quality of their work. In this paper, we ask how do the relatively well or poorly-performing crowds, working under specific task conditions, actually look like in terms of worker characteristics, such as demographics or personality traits. Our findings show that the face of a crowd is in fact indicative of the quality of their work.
KW - crowdsourcing
KW - demographics
KW - personality traits
KW - worker accuracy
UR - http://www.scopus.com/inward/record.url?scp=84871089459&partnerID=8YFLogxK
U2 - 10.1145/2396761.2398697
DO - 10.1145/2396761.2398697
M3 - Conference contribution
AN - SCOPUS:84871089459
SN - 9781450311564
T3 - ACM International Conference Proceeding Series
SP - 2583
EP - 2586
BT - CIKM 2012 - Proceedings of the 21st ACM International Conference on Information and Knowledge Management
Y2 - 29 October 2012 through 2 November 2012
ER -