Strojny Paweł, Kapela Ksawery, Lipp Natalia, Sikström Sverker
Faculty of Management and Social Communication, Jagiellonian University, Krakow, Poland.
Sano Centre for Computational Medicine, International Research Foundation, Kraków, Poland.
JMIR Serious Games. 2024 Dec 31;12:e56663. doi: 10.2196/56663.
Words are a natural way to describe mental states in humans, while numerical values are a convenient and effective way to carry out quantitative psychological research. With the growing interest of researchers in gaming disorder, the number of screening tools is growing. However, they all require self-quantification of mental states. The rapid development of natural language processing creates an opportunity to supplement traditional rating scales with a question-based computational language assessment approach that gives a deeper understanding of the studied phenomenon without losing the rigor of quantitative data analysis.
The aim of the study was to investigate whether transformer-based language model analysis of text responses from active gamers is a potential supplement to traditional rating scales. We compared a tool consisting of 4 open-ended questions formulated based on the clinician's intuition (not directly related to any existing rating scales for measuring gaming disorders) with the results of one of the commonly used rating scales.
Participants recruited using an online panel were asked to answer the Word-Based Gaming Disorder Test, consisting of 4 open-ended questions about gaming. Subsequently, they completed a closed-ended Gaming Disorders Test based on a numerical scale. Of the initial 522 responses collected, we removed a total of 105 due to 1 of 3 criteria (suspiciously low survey completion time, providing nonrelevant or incomplete responses). Final analyses were conducted on the responses of 417 participants. The responses to the open-ended questions were vectorized using HerBERT, a large language model based on Google's Bidirectional Encoder Representations from Transformers (BERT). Last, a machine learning model, specifically ridge regression, was used to predict the scores of the Gaming Disorder Test based on the features of the vectorized open-ended responses.
The Pearson correlation between the observable scores from the Gaming Disorder test and the predictions made by the model was 0.476 when using the answers of the 4 respondents as features. When using only 1 of the 4 text responses, the correlation ranged from 0.274 to 0.406.
Short open responses analyzed using natural language processing can contribute to a deeper understanding of gaming disorder at no additional cost in time. The obtained results confirmed 2 of 3 preregistered hypotheses. The written statements analyzed using the results of the model correlated with the rating scale. Furthermore, the inclusion in the model of data from more responses that take into account different perspectives on gaming improved the performance of the model. However, there is room for improvement, especially in terms of supplementing the questions with content that corresponds more directly to the definition of gaming disorder.
OSF Registries osf.io/957nz; https://osf.io/957nz.
词汇是描述人类心理状态的自然方式，而数值是进行定量心理学研究的便捷有效方式。随着研究人员对游戏障碍的兴趣日益浓厚，筛查工具的数量也在增加。然而，它们都需要对心理状态进行自我量化。自然语言处理的快速发展为用基于问题的计算语言评估方法补充传统评分量表创造了机会，这种方法能在不损失定量数据分析严谨性的情况下，更深入地理解所研究的现象。
本研究的目的是调查基于Transformer的语言模型对活跃游戏玩家文本回复的分析是否是传统评分量表的潜在补充。我们将一个由基于临床医生直觉制定的4个开放式问题组成的工具（与任何现有的游戏障碍测量评分量表无直接关联）与一种常用评分量表的结果进行了比较。
通过在线面板招募的参与者被要求回答基于词汇的游戏障碍测试，该测试由4个关于游戏的开放式问题组成。随后，他们完成了基于数字量表的封闭式游戏障碍测试。在最初收集的522份回复中，由于3个标准中的1个（调查完成时间异常短、提供不相关或不完整的回复），我们总共删除了105份。对417名参与者的回复进行了最终分析。使用HerBERT（一种基于谷歌的双向编码器表征变换器（BERT）的大语言模型）对开放式问题的回复进行向量化。最后，使用机器学习模型，特别是岭回归，根据向量化开放式回复的特征预测游戏障碍测试的分数。
当使用4名受访者的答案作为特征时，游戏障碍测试的可观察分数与模型预测之间的皮尔逊相关系数为0.476。当仅使用4个文本回复中的1个时，相关系数范围为0.274至0.406。
使用自然语言处理分析简短的开放式回复可以在不增加时间成本的情况下，有助于更深入地理解游戏障碍。获得的结果证实了预先注册的3个假设中的2个。使用模型结果分析的书面陈述与评分量表相关。此外，将考虑到对游戏不同观点的更多回复的数据纳入模型中，提高了模型的性能。然而，仍有改进的空间，特别是在补充与游戏障碍定义更直接对应的内容方面。
OSF注册中心osf.io/957nz；https://osf.io/957nz。
