Graduate Institute of Journalism, National Taiwan University, Taipei, Taiwan (R.O.C.).
Communication, Journalism, & Media Department, Suffolk University, Boston, Massachusetts, United States of America.
PLoS One. 2020 Oct 22;15(10):e0241045. doi: 10.1371/journal.pone.0241045. eCollection 2020.
The identification of bots is an important and complicated task. The bot classifier "Botometer" was successfully introduced as a way to estimate the number of bots in a given list of accounts and, as a consequence, has been frequently used in academic publications. Given its relevance for academic research and our understanding of the presence of automated accounts in any given Twitter discourse, we are interested in Botometer's diagnostic ability over time. To do so, we collected the Botometer scores for five datasets (three verified as bots, two verified as human; n = 4,134) in two languages (English/German) over three months. We show that the Botometer scores are imprecise when it comes to estimating bots; especially in a different language. We further show in an analysis of Botometer scores over time that Botometer's thresholds, even when used very conservatively, are prone to variance, which, in turn, will lead to false negatives (i.e., bots being classified as humans) and false positives (i.e., humans being classified as bots). This has immediate consequences for academic research as most studies in social science using the tool will unknowingly count a high number of human users as bots and vice versa. We conclude our study with a discussion about how computational social scientists should evaluate machine learning systems that are developed for identifying bots.
识别机器人是一项重要而复杂的任务。机器人分类器“Botometer”已被成功引入，用于估计给定账户列表中的机器人数量，因此在学术出版物中经常被使用。鉴于其对学术研究的重要性以及我们对任何给定 Twitter 话语中自动化账户的存在的理解，我们对 Botometer 的诊断能力随时间的变化感兴趣。为此，我们在三个月内收集了五个数据集（三个经证实为机器人，两个经证实为人；n=4134）的 Botometer 分数，这两个数据集分别使用两种语言（英语/德语）。我们表明，Botometer 分数在估计机器人时不够精确；尤其是在不同的语言环境中。我们进一步在 Botometer 分数随时间的分析中表明，Botometer 的阈值，即使使用非常保守的方式，也容易出现变化，这反过来又会导致假阴性（即机器人被错误地归类为人类）和假阳性（即人类被错误地归类为机器人）。这对学术研究产生了直接影响，因为使用该工具的大多数社会科学研究将无意识地将大量的人类用户错误地归类为机器人，反之亦然。我们在讨论中结束了我们的研究，讨论了计算社会科学家应该如何评估为识别机器人而开发的机器学习系统。
