Multifactorial and Complex Diseases Research Area, Bambino Gesù Children's Hospital, IRCCS, Rome, Italy.
Vaccine Research Department, FISABIO-Public Health, Valencia, Spain.
Front Public Health. 2022 Jul 29;10:948880. doi: 10.3389/fpubh.2022.948880. eCollection 2022.
Social media is increasingly being used to express opinions and attitudes toward vaccines. The vaccine stance of social media posts can be classified in almost real-time using machine learning. We describe the use of a Transformer-based machine learning model for analyzing vaccine stance of Italian tweets, and demonstrate the need to address changes over time in vaccine-related language, through periodic model retraining. Vaccine-related tweets were collected through a platform developed for the European Joint Action on Vaccination. Two datasets were collected, the first between November 2019 and June 2020, the second from April to September 2021. The tweets were manually categorized by three independent annotators. After cleaning, the total dataset consisted of 1,736 tweets with 3 categories (promotional, neutral, and discouraging). The manually classified tweets were used to train and test various machine learning models. The model that classified the data most similarly to humans was XLM-Roberta-large, a multilingual version of the Transformer-based model RoBERTa. The model hyper-parameters were tuned and then the model ran five times. The fine-tuned model with the best F-score over the validation dataset was selected. Running the selected fine-tuned model on just the first test dataset resulted in an accuracy of 72.8% (F-score 0.713). Using this model on the second test dataset resulted in a 10% drop in accuracy to 62.1% (F-score 0.617), indicating that the model recognized a difference in language between the datasets. On the combined test datasets the accuracy was 70.1% (F-score 0.689). Retraining the model using data from the first and second datasets increased the accuracy over the second test dataset to 71.3% (F-score 0.713), a 9% improvement from when using just the first dataset for training. The accuracy over the first test dataset remained the same at 72.8% (F-score 0.721). The accuracy over the combined test datasets was then 72.4% (F-score 0.720), a 2% improvement. Through fine-tuning a machine-learning model on task-specific data, the accuracy achieved in categorizing tweets was close to that expected by a single human annotator. Regular training of machine-learning models with recent data is advisable to maximize accuracy.
社交媒体越来越多地被用于表达对疫苗的看法和态度。使用机器学习可以近乎实时地对社交媒体帖子的疫苗立场进行分类。我们描述了一种基于转换器的机器学习模型在分析意大利推文疫苗立场方面的应用，并通过定期模型重新训练证明了有必要解决与疫苗相关的语言随时间的变化。通过为欧洲联合疫苗行动开发的一个平台收集了与疫苗相关的推文。收集了两个数据集，第一个数据集于 2019 年 11 月至 2020 年 6 月之间收集，第二个数据集于 2021 年 4 月至 9 月之间收集。这些推文由三名独立的注释者手动分类。经过清理，总数据集由 1736 条推文组成，分为 3 类（宣传、中立和劝阻）。手动分类的推文用于训练和测试各种机器学习模型。与人类最相似地分类数据的模型是 XLM-Roberta-large，这是一种基于转换器的模型 RoBERTa 的多语言版本。调整了模型超参数，然后模型运行了 5 次。选择在验证数据集上具有最佳 F 分数的微调模型。仅在第一个测试数据集上运行选择的微调模型，其准确率为 72.8%（F 分数 0.713）。在第二个测试数据集上使用此模型导致准确率下降 10%，降至 62.1%（F 分数 0.617），表明模型识别出数据集之间语言的差异。在联合测试数据集中，准确率为 70.1%（F 分数 0.689）。使用来自第一个和第二个数据集的数据重新训练模型，使模型在第二个测试数据集上的准确率提高到 71.3%（F 分数 0.713），比仅使用第一个数据集进行训练提高了 9%。在第一个测试数据集上的准确率保持不变，为 72.8%（F 分数 0.721）。然后，联合测试数据集的准确率为 72.4%（F 分数 0.720），提高了 2%。通过在特定于任务的数据上微调机器学习模型，可以实现接近单个人类注释者的分类推文的准确性。建议定期用最新数据训练机器学习模型，以最大限度地提高准确性。
