TY - GEN
T1 - Overview of the CLEF-2018 checkthat! lab on automatic identification and verification of political claims
AU - Nakov, Preslav
AU - Barrón-Cedeño, Alberto
AU - Elsayed, Tamer
AU - Suwaileh, Reem
AU - Màrquez, Lluís
AU - Zaghouani, Wajdi
AU - Atanasova, Pepa
AU - Kyuchukov, Spas
AU - Da San Martino, Giovanni
N1 - Publisher Copyright:
© Springer Nature Switzerland AG 2018.
PY - 2018
Y1 - 2018
N2 - We present an overview of the CLEF-2018 CheckThat! Lab on Automatic Identification and Verification of Political Claims. In its starting year, the lab featured two tasks. Task 1 asked to predict which (potential) claims in a political debate should be prioritized for fact-checking; in particular, given a debate or a political speech, the goal was to produce a ranked list of its sentences based on their worthiness for fact-checking. Task 2 asked to assess whether a given check-worthy claim made by a politician in the context of a debate/speech is factually true, half-true, or false. We offered both tasks in English and in Arabic. In terms of data, for both tasks, we focused on debates from the 2016 US Presidential Campaign, as well as on some speeches during and after the campaign (we also provided translations in Arabic), and we relied on comments and factuality judgments from factcheck.org and snopes.com, which we further refined manually. A total of 30 teams registered to participate in the lab, and 9 of them actually submitted runs. The evaluation results show that the most successful approaches used various neural networks (esp. for Task 1) and evidence retrieval from the Web (esp. for Task 2). We release all datasets, the evaluation scripts, and the submissions by the participants, which should enable further research in both check-worthiness estimation and automatic claim verification.
AB - We present an overview of the CLEF-2018 CheckThat! Lab on Automatic Identification and Verification of Political Claims. In its starting year, the lab featured two tasks. Task 1 asked to predict which (potential) claims in a political debate should be prioritized for fact-checking; in particular, given a debate or a political speech, the goal was to produce a ranked list of its sentences based on their worthiness for fact-checking. Task 2 asked to assess whether a given check-worthy claim made by a politician in the context of a debate/speech is factually true, half-true, or false. We offered both tasks in English and in Arabic. In terms of data, for both tasks, we focused on debates from the 2016 US Presidential Campaign, as well as on some speeches during and after the campaign (we also provided translations in Arabic), and we relied on comments and factuality judgments from factcheck.org and snopes.com, which we further refined manually. A total of 30 teams registered to participate in the lab, and 9 of them actually submitted runs. The evaluation results show that the most successful approaches used various neural networks (esp. for Task 1) and evidence retrieval from the Web (esp. for Task 2). We release all datasets, the evaluation scripts, and the submissions by the participants, which should enable further research in both check-worthiness estimation and automatic claim verification.
KW - Check-worthiness estimation
KW - Computational journalism
KW - Fact-checking
KW - Veracity
UR - http://www.scopus.com/inward/record.url?scp=85051084265&partnerID=8YFLogxK
U2 - 10.1007/978-3-319-98932-7_32
DO - 10.1007/978-3-319-98932-7_32
M3 - Conference contribution
AN - SCOPUS:85051084265
SN - 9783319989310
T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
SP - 372
EP - 387
BT - Experimental IR Meets Multilinguality, Multimodality, and Interaction - 9th International Conference of the CLEF Association, CLEF 2018, Proceedings
A2 - SanJuan, Eric
A2 - Murtagh, Fionn
A2 - Nie, Jian Yun
A2 - Soulier, Laure
A2 - Cappellato, Linda
A2 - Bellot, Patrice
A2 - Mothe, Josiane
A2 - Trabelsi, Chiraz
A2 - Ferro, Nicola
PB - Springer Verlag
T2 - 9th International Conference of the CLEF Association, CLEF 2018
Y2 - 10 September 2018 through 14 September 2018
ER -