Gu Juan, Liu Jiali, Zeng Lijuan, Yu Yiqing, Qiu Yufei, Yue Yake, Tong Mengjie, Yang Fen, Zhang Xiaohong
College of Nursing, Hubei University of Chinese Medicine, Wuhan, 430065, Hubei Province, China.
Hubei Shizhen Laboratory, Wuhan, 430065, Hubei Province, China.
Sci Rep. 2025 Jul 1;15(1):20621. doi: 10.1038/s41598-025-06358-2.
The capability of ChatGPT to understand and generate human-readable text has prompted the investigation of its potential as mental health assessment tools. This study aims to explore the validity of ChatGPT in assessing loneliness and online social support among college students by comparing scoring consistency between ChatGPT and the validated questionnaires. This was a cross-sectional study between June and August 2024. We pre-trained ChatGPT-4 based on the validated University of California Los Angeles Loneliness Scale-6 (ULS-6) and Chinese Youth Version of the Online Social Support Scale (OSSS-CS), creating a structured interview questionnaire. Participants were invited to complete both the ChatGPT-created questionnaire and the validated questionnaires. We used Spearman correlation analysis, Intra-class correlation coefficients (ICC), and Bland-Altman plots to assess the agreement between the scores from ChatGPT and the validated questionnaires. In addition, we evaluated ceiling and floor effects. A total of 216 college students participated the survey. The results demonstrated a good consistency between the scores obtained from ChatGPT and the validated questionnaires, with ICC of 0.81 (95% CI 0.75-0.85, p < 0.001) for ULS-6 and 0.95 (95% CI 0.94-0.96, p < 0.001) for OSSS-CS. The Spearman correlation coefficients were 0.64 (p < 0.001) for ULS-6 and 0.89 (p < 0.001) for OSSS-CS, indicating a moderate correlation. No ceiling or floor effects were observed. The ChatGPT-created questionnaire demonstrated acceptable consistency with the validated questionnaires. Future studies can further explore the performance of ChatGPT in different populations and domains, as well as how to integrate it with validated questionnaires to enhance the accessibility of assessments.
ChatGPT理解和生成人类可读文本的能力促使人们对其作为心理健康评估工具的潜力进行研究。本研究旨在通过比较ChatGPT与经过验证的问卷之间的评分一致性，探讨ChatGPT在评估大学生孤独感和在线社交支持方面的有效性。这是一项于2024年6月至8月进行的横断面研究。我们基于经过验证的加利福尼亚大学洛杉矶分校孤独感量表-6（ULS-6）和中国青年版在线社交支持量表（OSSS-CS）对ChatGPT-4进行预训练，创建了一份结构化访谈问卷。邀请参与者同时完成ChatGPT创建的问卷和经过验证的问卷。我们使用Spearman相关性分析、组内相关系数（ICC）和Bland-Altman图来评估ChatGPT的评分与经过验证的问卷之间的一致性。此外，我们还评估了天花板效应和地板效应。共有216名大学生参与了调查。结果表明，ChatGPT获得的评分与经过验证的问卷之间具有良好的一致性，ULS-6的ICC为0.81（95%CI 0.75-0.85，p<0.001），OSSS-CS的ICC为0.95（95%CI 0.94-0.96，p<0.001）。ULS-6的Spearman相关系数为0.64（p<0.001），OSSS-CS的Spearman相关系数为0.89（p<0.001），表明存在中度相关性。未观察到天花板效应或地板效应。ChatGPT创建的问卷与经过验证的问卷表现出可接受的一致性。未来的研究可以进一步探索ChatGPT在不同人群和领域的表现，以及如何将其与经过验证的问卷相结合以提高评估的可及性。
