TY - GEN
T1 - ArTest
T2 - 43rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2020
AU - Hasanain, Maram
AU - Barkallah, Yassmine
AU - Suwaileh, Reem
AU - Kutlu, Mucahid
AU - Elsayed, Tamer
N1 - Publisher Copyright:
© 2020 ACM.
PY - 2020/7/25
Y1 - 2020/7/25
N2 - The scarcity of Arabic test collections has long hindered information retrieval (IR) research over the Arabic Web. In this work, we present ArTest, the first large-scale test collection designed for the evaluation of ad-hoc search over the Arabic Web. ArTest uses ArabicWeb16, a collection of around 150M Arabic Web pages as the document collection, and includes 50 topics, 10,529 relevance judgments, and (more importantly) a rationale behind each judgment. To our knowledge, this is also the first IR test collection that includes rationales of primary assessors (i.e., topic developers) for their relevance judgments, exhibiting a useful resource for understanding the relevance phenomena. Finally, ArTest is made publicly-available for the research community.
AB - The scarcity of Arabic test collections has long hindered information retrieval (IR) research over the Arabic Web. In this work, we present ArTest, the first large-scale test collection designed for the evaluation of ad-hoc search over the Arabic Web. ArTest uses ArabicWeb16, a collection of around 150M Arabic Web pages as the document collection, and includes 50 topics, 10,529 relevance judgments, and (more importantly) a rationale behind each judgment. To our knowledge, this is also the first IR test collection that includes rationales of primary assessors (i.e., topic developers) for their relevance judgments, exhibiting a useful resource for understanding the relevance phenomena. Finally, ArTest is made publicly-available for the research community.
KW - ad-hoc search
KW - evaluation
KW - less-resourced language
KW - retrieval
UR - http://www.scopus.com/inward/record.url?scp=85090111113&partnerID=8YFLogxK
U2 - 10.1145/3397271.3401223
DO - 10.1145/3397271.3401223
M3 - Conference contribution
AN - SCOPUS:85090111113
T3 - SIGIR 2020 - Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval
SP - 2017
EP - 2020
BT - SIGIR 2020 - Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval
PB - Association for Computing Machinery, Inc
Y2 - 25 July 2020 through 30 July 2020
ER -