Mork Tiril Egset, Mjøs Håkon Garnes, Nilsen Harald Giskegjerde, Kjelsrud Sindre, Lundervold Alexander Selvikvåg, Lundervold Arvid, Jammer Ib
Det medisinske fakultet, Universitetet i Bergen.
Høgskulen på Vestlandet.
Tidsskr Nor Laegeforen. 2025 Feb 10;145(2). doi: 10.4045/tidsskr.24.0402. Print 2025 Feb 11.
Several studies have investigated how large language models answer health-related questions. In a study from 2023, responses to health-related questions in English generated by the language model GPT-3.5 were perceived as more empathetic and informative than responses from doctors. We wanted to apply the newer language model GPT-4 in Norwegian to investigate how respondents with a healthcare background rated responses to health-related questions from doctors and those generated by the language model.
A total of 192 health-related questions with corresponding answers from doctors were sourced from the website Studenterspør.no. The language model GPT-4 was used to generate a new set of answers to the same questions. Both sets of answers were evaluated by 344 respondents with a background in health care. The respondents, who were blinded to whether the answer was generated by a doctor or the language model, were asked to rate the empathy, quality of information and helpfulness of the answers.
The survey consisted of 344 respondents and 192 questions. The average number of evaluations per answer was 5.7. There was a significant difference between doctors' answers and those generated by GPT-4 in terms of perceived empathy (p < 0.001), quality of information (p < 0.001) and helpfulness (p < 0.001).
The answers generated by GPT-4 were rated as more empathetic, informative and helpful than the answers from doctors. This suggests that AI could serve as an aid to healthcare personnel by drafting good responses to health-related questions.
多项研究调查了大语言模型如何回答与健康相关的问题。在一项2023年的研究中，语言模型GPT - 3.5生成的英文健康相关问题的回答被认为比医生的回答更具同理心和信息量。我们希望在挪威语环境中应用更新的语言模型GPT - 4，以调查具有医疗保健背景的受访者如何评价医生和语言模型针对健康相关问题给出的回答。
从网站Studenter.spør.no上获取了总共192个与健康相关的问题以及医生给出的相应答案。使用语言模型GPT - 4对相同问题生成一组新的答案。两组答案由344名具有医疗保健背景的受访者进行评估。受访者不知道答案是由医生还是语言模型生成的，他们被要求对答案的同理心、信息质量和有用性进行评分。
该调查包括344名受访者和192个问题。每个答案的平均评估次数为5.7次。在感知到的同理心（p < 0.001）、信息质量（p < 0.001）和有用性（p < 0.001）方面，医生的答案与GPT - 4生成的答案之间存在显著差异。
GPT - 4生成的答案在同理心、信息量和有用性方面的评分高于医生的答案。这表明人工智能可以通过起草针对健康相关问题的良好回答，为医护人员提供帮助。
