Radboud University, Erasmusplein 1, Nijmegen, 6525, HT, The Netherlands.
Vrije Universiteit Amsterdam, De Boelelaan 1111, Amsterdam, 1081, HV, The Netherlands.
BMC Med Inform Decis Mak. 2020 Feb 18;20(1):33. doi: 10.1186/s12911-020-1046-y.
We developed a system to automatically classify stance towards vaccination in Twitter messages, with a focus on messages with a negative stance. Such a system makes it possible to monitor the ongoing stream of messages on social media, offering actionable insights into public hesitance with respect to vaccination. At the moment, such monitoring is done by means of regular sentiment analysis with a poor performance on detecting negative stance towards vaccination. For Dutch Twitter messages that mention vaccination-related key terms, we annotated their stance and feeling in relation to vaccination (provided that they referred to this topic). Subsequently, we used these coded data to train and test different machine learning set-ups. With the aim to best identify messages with a negative stance towards vaccination, we compared set-ups at an increasing dataset size and decreasing reliability, at an increasing number of categories to distinguish, and with different classification algorithms.
We found that Support Vector Machines trained on a combination of strictly and laxly labeled data with a more fine-grained labeling yielded the best result, at an F1-score of 0.36 and an Area under the ROC curve of 0.66, considerably outperforming the currently used sentiment analysis that yielded an F1-score of 0.25 and an Area under the ROC curve of 0.57. We also show that the recall of our system could be optimized to 0.60 at little loss of precision.
The outcomes of our study indicate that stance prediction by a computerized system only is a challenging task. Nonetheless, the model showed sufficient recall on identifying negative tweets so as to reduce the manual effort of reviewing messages. Our analysis of the data and behavior of our system suggests that an approach is needed in which the use of a larger training dataset is combined with a setting in which a human-in-the-loop provides the system with feedback on its predictions.
我们开发了一种自动分类推特消息中疫苗接种立场的系统，重点关注具有负面立场的消息。这样的系统使得监测社交媒体上正在进行的消息流成为可能，为公众对疫苗接种的犹豫提供了可操作的见解。目前，这种监测是通过定期的情感分析来进行的，但在检测对疫苗接种的负面立场方面表现不佳。对于提到与疫苗接种相关关键词的荷兰推特消息，我们对它们与疫苗接种相关的立场和感受进行了注释（前提是它们提到了这个话题）。随后，我们使用这些编码数据来训练和测试不同的机器学习设置。为了最好地识别对疫苗接种持负面立场的消息，我们比较了在数据集大小增加、可靠性降低、类别数量增加和使用不同分类算法的情况下的设置。
我们发现，支持向量机在严格和宽松标记数据的组合上进行训练，并使用更细粒度的标记，产生了最好的结果，F1 得分为 0.36，ROC 曲线下面积为 0.66，明显优于目前使用的情感分析，F1 得分为 0.25，ROC 曲线下面积为 0.57。我们还表明，可以在不损失精度的情况下，将系统的召回率优化到 0.60。
我们的研究结果表明，仅通过计算机系统进行立场预测是一项具有挑战性的任务。尽管如此，该模型在识别负面推文方面表现出了足够的召回率，从而减少了对消息进行人工审查的工作量。我们对数据的分析和系统的行为表明，需要一种方法，将使用更大的训练数据集与一种设置相结合，在这种设置中，人工为系统提供对其预测的反馈。
