Guckenberger Matthias, Andratschke Nicolaus, Ahmadsei Maiwand, Christ Sebastian Matthias, Heusel Astrid Elisabeth, Kamal Sandeep, Kroese Tiuri Ewout, Looman Esmée Lauren, Reichl Sabrina, Vlaskou Badra Eugenia, von der Grün Jens, Willmann Jonas, Tanadini-Lang Stephanie, Mayinger Michael
Department of Radiation Oncology, University Hospital of Zurich, University of Zurich, Zurich, Switzerland.
Department of Radiation Oncology, University Hospital of Zurich, University of Zurich, Zurich, Switzerland.
Radiother Oncol. 2023 Nov;188:109894. doi: 10.1016/j.radonc.2023.109894. Epub 2023 Sep 1.
To evaluate the potential of the artificial intelligence (AI) chatbot ChatGPT in supporting young clinical scientists with scientific tasks in radio oncological research.
Seven scientific tasks were to be completed in 3 h by 8 radiation oncologists with different scientific experience working at a university hospital: creation of a scientific synopsis, creation of a research question and corresponding clinical trial hypotheses, writing of the first paragraph of a manuscript introduction, clinical trial sample size calculation, and clinical data analyses (multivariate analysis, boxplot and survival curve). No participant had prior experience with an AI chatbot. All participants were instructed in ChatGPT v3.5 and its use was provided for all tasks. Answers were scored independently by two blinded experts. The subjective value of ChatGPT was rated by each participant. Data were analyzed with regression-, t-test and Spearman correlation (p < 0.05).
Participants completed tasks 1-3 with an average score of 50% and 4-7 with 56%. Scientific experience, number of original publications and of first/last authorships showed a positive correlation with overall scoring (p = 0.01-0.04). Participants with little to moderate scientific experience scored ChatGPT to be more helpful in solving tasks 4-7 compared to more experienced participants (p = 0.04), with simultaneously presenting lower scorings (p = 0.03).
ChatGPT did not compensate for differences in scientific experience of young clinical scientists, with less experienced researchers believing false AI-generated scientific results.
评估人工智能（AI）聊天机器人ChatGPT在支持年轻临床科学家进行放射肿瘤学研究中的科学任务方面的潜力。
8名在大学医院工作、具有不同科学经验的放射肿瘤学家要在3小时内完成7项科学任务：创建科学概要、提出研究问题和相应的临床试验假设、撰写手稿引言的第一段、计算临床试验样本量以及进行临床数据分析（多变量分析、箱线图和生存曲线）。没有参与者此前有使用AI聊天机器人的经验。所有参与者都接受了ChatGPT v3.5的指导，并在所有任务中都可使用它。答案由两名不知情的专家独立评分。每位参与者对ChatGPT的主观价值进行了评分。使用回归分析、t检验和Spearman相关性分析数据（p<0.05）。
参与者完成任务1 - 3的平均得分为50%，完成任务4 - 7的平均得分为56%。科学经验、原始出版物数量以及第一/最后作者身份与总体评分呈正相关（p = 0.01 - 0.04）。与经验更丰富的参与者相比，科学经验较少至中等的参与者认为ChatGPT在解决任务4 - 7方面更有帮助（p = 0.04），但同时得分较低（p = 0.03）。
ChatGPT无法弥补年轻临床科学家在科学经验上的差异，经验较少的研究人员会相信由AI生成的虚假科学结果。
