Department of Psychology, Beijing Forestry University, Beijing, China.
Front Public Health. 2024 Jul 8;12:1401322. doi: 10.3389/fpubh.2024.1401322. eCollection 2024.
Implementing machine learning prediction of negative attitudes towards suicide may improve health outcomes. However, in previous studies, varied forms of negative attitudes were not adequately considered, and developed models lacked rigorous external validation. By analyzing a large-scale social media dataset (Sina Weibo), this paper aims to fully cover varied forms of negative attitudes and develop a classification model for predicting negative attitudes as a whole, and then to externally validate its performance on population and individual levels.
938,866 Weibo posts with relevant keywords were downloaded, including 737,849 posts updated between 2009 and 2014 (), and 201,017 posts updated between 2015 and 2020 (). (1) For model development, based on 10,000 randomly selected posts from , a human-based content analysis was performed to manually determine labels of each post (non-negative or negative attitudes). Then, a computer-based content analysis was conducted to automatically extract psycholinguistic features from each of the same 10,000 posts. Finally, a classification model for predicting negative attitudes was developed on selected features. (2) For model validation, on the population level, the developed model was implemented on remaining 727,849 posts from , and was externally validated by comparing proportions of negative attitudes between predicted and human-coded results. Besides, on the individual level, similar analyses were performed on 300 randomly selected posts from , and the developed model was externally validated by comparing labels of each post between predicted and actual results.
For model development, the F1 and area under ROC curve (AUC) values reached 0.93 and 0.97. For model validation, on the population level, significant differences but very small effect sizes were observed for the whole sample ( = 32.35, < 0.001; Cramer's V = 0.007, < 0.001), men ( = 9.48, = 0.002; Cramer's V = 0.005, = 0.002), and women ( = 25.34, < 0.001; Cramer's V = 0.009, < 0.001). Besides, on the individual level, the F1 and AUC values reached 0.76 and 0.74.
This study demonstrates the efficiency and necessity of machine learning prediction of negative attitudes as a whole, and confirms that external validation is essential before implementing prediction models into practice.
实施机器学习预测对自杀的消极态度可能会改善健康结果。然而，在之前的研究中，不同形式的消极态度没有得到充分考虑，而且开发的模型缺乏严格的外部验证。通过分析大规模的社交媒体数据集（新浪微博），本文旨在充分涵盖不同形式的消极态度，并开发一种整体预测消极态度的分类模型，然后在人群和个体水平上对其性能进行外部验证。
下载了 938866 条包含相关关键词的微博帖子，其中 737849 条帖子更新于 2009 年至 2014 年（），201017 条帖子更新于 2015 年至 2020 年（）。（1）为了进行模型开发，我们从随机选择的 10000 个帖子中，基于人工的内容分析手动确定每个帖子的标签（非消极或消极态度）。然后，我们基于计算机的内容分析从相同的 10000 个帖子中自动提取心理语言特征。最后，我们在选定的特征上开发了一种预测消极态度的分类模型。（2）为了进行模型验证，在人群水平上，我们将开发的模型应用于剩余的 727849 个来自的帖子，并通过比较预测结果和人工编码结果之间的消极态度比例来进行外部验证。此外，在个体水平上，我们对来自的 300 个随机选择的帖子进行了类似的分析，并通过比较预测结果和实际结果之间的每个帖子的标签来进行外部验证。
对于模型开发，F1 和 ROC 曲线下面积（AUC）值分别达到 0.93 和 0.97。对于模型验证，在人群水平上，对于整个样本（ = 32.35， < 0.001；Cramer's V = 0.007， < 0.001）、男性（ = 9.48， = 0.002；Cramer's V = 0.005， = 0.002）和女性（ = 25.34， < 0.001；Cramer's V = 0.009， < 0.001），都观察到了显著差异，但效应量很小。此外，在个体水平上，F1 和 AUC 值分别达到 0.76 和 0.74。
本研究证明了整体预测消极态度的机器学习的效率和必要性，并证实了在将预测模型应用于实践之前，外部验证是必要的。
