Moylan Kayley, Doherty Kevin
School of Information and Communication Studies, University College Dublin, Dublin, Ireland.
J Med Internet Res. 2025 Apr 25;27:e67114. doi: 10.2196/67114.
Recent years have seen an immense surge in the creation and use of chatbots as social and mental health companions. Aiming to provide empathic responses in support of the delivery of personalized support, these tools are often presented as offering immense potential. However, it is also essential that we understand the risks of their deployment, including their potential adverse impacts on the mental health of users, including those most at risk.
The study aims to assess the ethical and pragmatic clinical implications of using chatbots that claim to aid mental health. While several studies within human-computer interaction and related fields have examined users' perceptions of such systems, few studies have engaged mental health professionals in critical analysis of their conduct as mental health support tools. This paper comprises, in turn, an effort to assess the ethical and pragmatic clinical implications of using chatbots that claim to aid mental health.
This study included 8 interdisciplinary mental health professional participants (from psychology and psychotherapy to social care and crisis volunteer workers) in a mixed methods and hands-on analysis of 2 popular mental health-related chatbots' data handling, interface design, and responses. This analysis was carried out through profession-specific tasks with each chatbot, eliciting participants' perceptions through both the Trust in Automation scale and semistructured interviews. Through thematic analysis and a 2-tailed, paired t test, these chatbots' implications for mental health support were thus evaluated.
Qualitative analysis revealed emphatic initial impressions among mental health professionals of chatbot responses likely to produce harm, exhibiting a generic mode of care, and risking user dependence and manipulation given the central role of trust in the therapeutic relationship. Trust scores from the Trust in Automation scale, while exhibiting no statistically significant differences between the chatbots (t=-0.76; P=.48), indicated medium to low trust scores for each chatbot. The findings of this work highlight that the design and development of artificial intelligence (AI)-driven mental health-related solutions must be undertaken with utmost caution. The mental health professionals in this study collectively resist these chatbots and make clear that AI-driven chatbots used for mental health by at-risk users invite several potential and specific harms.
Through this work, we contributed insights into the mental health professional perspective on the design of chatbots used for mental health and underscore the necessity of ongoing critical assessment and iterative refinement to maximize the benefits and minimize the risks associated with integrating AI into mental health support.
近年来，聊天机器人作为社交和心理健康伙伴的创建和使用激增。这些工具旨在提供共情回应以支持个性化支持的提供，通常被认为具有巨大潜力。然而，我们理解其部署风险也至关重要，包括它们对用户心理健康的潜在不利影响，尤其是对那些风险最高的用户。
本研究旨在评估使用声称有助于心理健康的聊天机器人的伦理和务实临床意义。虽然人机交互及相关领域的多项研究考察了用户对这类系统的看法，但很少有研究让心理健康专业人员对其作为心理健康支持工具的行为进行批判性分析。本文依次致力于评估使用声称有助于心理健康的聊天机器人的伦理和务实临床意义。
本研究纳入了8名跨学科心理健康专业参与者（从心理学和心理治疗到社会护理和危机志愿者工作者），采用混合方法并亲自动手分析2个流行的与心理健康相关的聊天机器人的数据处理、界面设计和回复。通过针对每个聊天机器人的特定专业任务进行此分析，通过自动化信任量表和半结构化访谈引出参与者的看法。通过主题分析和双尾配对t检验，评估这些聊天机器人对心理健康支持的影响。
定性分析揭示了心理健康专业人员对聊天机器人回复的初步强烈印象，这些回复可能造成伤害，呈现出一般化的护理模式，并且鉴于信任在治疗关系中的核心作用，存在导致用户依赖和被操纵的风险。自动化信任量表的信任分数虽然在聊天机器人之间没有显示出统计学上的显著差异（t = -0.76；P = 0.48），但表明每个聊天机器人的信任分数为中到低。这项工作的结果强调，人工智能驱动的与心理健康相关的解决方案的设计和开发必须极其谨慎。本研究中的心理健康专业人员集体抵制这些聊天机器人，并明确表示，有风险的用户使用人工智能驱动的聊天机器人进行心理健康服务会带来一些潜在的特定危害。
通过这项工作，我们提供了关于心理健康专业人员对用于心理健康的聊天机器人设计的观点的见解，并强调持续进行批判性评估和迭代改进的必要性，以最大限度地提高益处并最小化将人工智能整合到心理健康支持中的相关风险。
