Predicting negative attitudes towards suicide in social media texts: prediction model development and validation study.

BACKGROUND

Implementing machine learning prediction of negative attitudes towards suicide may improve health outcomes. However, in previous studies, varied forms of negative attitudes were not adequately considered, and developed models lacked rigorous external validation. By analyzing a large-scale social media dataset (Sina Weibo), this paper aims to fully cover varied forms of negative attitudes and develop a classification model for predicting negative attitudes as a whole, and then to externally validate its performance on population and individual levels.

METHODS

938,866 Weibo posts with relevant keywords were downloaded, including 737,849 posts updated between 2009 and 2014 (), and 201,017 posts updated between 2015 and 2020 (). (1) For model development, based on 10,000 randomly selected posts from , a human-based content analysis was performed to manually determine labels of each post (non-negative or negative attitudes). Then, a computer-based content analysis was conducted to automatically extract psycholinguistic features from each of the same 10,000 posts. Finally, a classification model for predicting negative attitudes was developed on selected features. (2) For model validation, on the population level, the developed model was implemented on remaining 727,849 posts from , and was externally validated by comparing proportions of negative attitudes between predicted and human-coded results. Besides, on the individual level, similar analyses were performed on 300 randomly selected posts from , and the developed model was externally validated by comparing labels of each post between predicted and actual results.

RESULTS

For model development, the F1 and area under ROC curve (AUC) values reached 0.93 and 0.97. For model validation, on the population level, significant differences but very small effect sizes were observed for the whole sample (  = 32.35,  < 0.001; Cramer's V = 0.007,  < 0.001), men (  = 9.48,  = 0.002; Cramer's V = 0.005,  = 0.002), and women (  = 25.34,  < 0.001; Cramer's V = 0.009,  < 0.001). Besides, on the individual level, the F1 and AUC values reached 0.76 and 0.74.

CONCLUSION

This study demonstrates the efficiency and necessity of machine learning prediction of negative attitudes as a whole, and confirms that external validation is essential before implementing prediction models into practice.