TY - GEN
T1 - Potentials of ChatGPT for Annotating Vaccine Related Tweets
AU - Biswas, Md Rafiul
AU - Mohsen, Farida
AU - Shah, Zubair
AU - Zaghouani, Wajdi
N1 - Publisher Copyright:
© 2023 IEEE.
PY - 2023/11
Y1 - 2023/11
N2 - This study evaluates ChatGPT's performance in annotating vaccine-related Arabic tweets by comparing its annotations with human annotations. A dataset of 2,100 tweets representing various factors contributing to vaccine hesitancy was examined. Two domain experts annotated the data, with a third resolving conflicts. ChatGPT was then employed to annotate the same dataset using specific prompts for each factor. The ChatGPT annotations were evaluated through zero-shot, one-shot, and few-shot learning tests, with an average accuracy of 82.14%, 83.85%, and 85.57%, respectively. Precision averaged around 86%, minimizing false positives. The average recall and F1-score ranged from 0.74 to 0.80 and 0.65 to 0.93, respectively. AUC for zero-shot, one-shot, and few-shot learning was 0.79, 0.80, and 0.83. In cases of ambiguity, both human annotators and ChatGPT faced challenges. These findings suggest that ChatGPT holds promise as a tool for annotating vaccine-related tweets.
AB - This study evaluates ChatGPT's performance in annotating vaccine-related Arabic tweets by comparing its annotations with human annotations. A dataset of 2,100 tweets representing various factors contributing to vaccine hesitancy was examined. Two domain experts annotated the data, with a third resolving conflicts. ChatGPT was then employed to annotate the same dataset using specific prompts for each factor. The ChatGPT annotations were evaluated through zero-shot, one-shot, and few-shot learning tests, with an average accuracy of 82.14%, 83.85%, and 85.57%, respectively. Precision averaged around 86%, minimizing false positives. The average recall and F1-score ranged from 0.74 to 0.80 and 0.65 to 0.93, respectively. AUC for zero-shot, one-shot, and few-shot learning was 0.79, 0.80, and 0.83. In cases of ambiguity, both human annotators and ChatGPT faced challenges. These findings suggest that ChatGPT holds promise as a tool for annotating vaccine-related tweets.
KW - ChatGPT
KW - Data annotation
KW - Twitter
KW - large language model
KW - vaccine hesitancy
UR - http://www.scopus.com/inward/record.url?scp=85183458277&partnerID=8YFLogxK
U2 - 10.1109/SNAMS60348.2023.10375433
DO - 10.1109/SNAMS60348.2023.10375433
M3 - Conference contribution
AN - SCOPUS:85183458277
T3 - Proceedings - 2023 10th International Conference on Social Networks Analysis, Management and Security, SNAMS 2023
BT - Proceedings - 2023 10th International Conference on Social Networks Analysis, Management and Security, SNAMS 2023
PB - Institute of Electrical and Electronics Engineers Inc.
T2 - 10th International Conference on Social Networks Analysis, Management and Security, SNAMS 2023
Y2 - 21 November 2023 through 24 November 2023
ER -