Division of Nephrology and Hypertension, Department of Medicine, Mayo Clinic, Rochester, Minnesota.
Clin J Am Soc Nephrol. 2024 Jan 1;19(1):35-43. doi: 10.2215/CJN.0000000000000330. Epub 2023 Oct 18.
ChatGPT is a novel tool that allows people to engage in conversations with an advanced machine learning model. ChatGPT's performance in the US Medical Licensing Examination is comparable with a successful candidate's performance. However, its performance in the nephrology field remains undetermined. This study assessed ChatGPT's capabilities in answering nephrology test questions.
Questions sourced from Nephrology Self-Assessment Program and Kidney Self-Assessment Program were used, each with multiple-choice single-answer questions. Questions containing visual elements were excluded. Each question bank was run twice using GPT-3.5 and GPT-4. Total accuracy rate, defined as the percentage of correct answers obtained by ChatGPT in either the first or second run, and the total concordance, defined as the percentage of identical answers provided by ChatGPT during both runs, regardless of their correctness, were used to assess its performance.
A comprehensive assessment was conducted on a set of 975 questions, comprising 508 questions from Nephrology Self-Assessment Program and 467 from Kidney Self-Assessment Program. GPT-3.5 resulted in a total accuracy rate of 51%. Notably, the employment of Nephrology Self-Assessment Program yielded a higher accuracy rate compared with Kidney Self-Assessment Program (58% versus 44%; P < 0.001). The total concordance rate across all questions was 78%, with correct answers exhibiting a higher concordance rate (84%) compared with incorrect answers (73%) ( P < 0.001). When examining various nephrology subfields, the total accuracy rates were relatively lower in electrolyte and acid-base disorder, glomerular disease, and kidney-related bone and stone disorders. The total accuracy rate of GPT-4's response was 74%, higher than GPT-3.5 ( P < 0.001) but remained below the passing threshold and average scores of nephrology examinees (77%).
ChatGPT exhibited limitations regarding accuracy and repeatability when addressing nephrology-related questions. Variations in performance were evident across various subfields.
ChatGPT 是一种新颖的工具，可让人们与先进的机器学习模型进行对话。ChatGPT 在 USMLE 中的表现可与成功考生的表现相媲美。然而，其在肾脏病学领域的表现尚不确定。本研究评估了 ChatGPT 回答肾脏病学试题的能力。
使用来源于肾脏病自我评估计划和肾脏自我评估计划的试题，每个题库均包含多项选择题单项答案。排除包含视觉元素的试题。每个题库均使用 GPT-3.5 和 GPT-4 运行两次。总准确率定义为 ChatGPT 在任一次运行中答对的试题百分比，总一致性定义为 ChatGPT 在两次运行中提供的相同答案的百分比，无论其是否正确，均用于评估其性能。
对 975 道试题进行了全面评估，其中 508 道试题来源于肾脏病自我评估计划，467 道试题来源于肾脏自我评估计划。GPT-3.5 的总准确率为 51%。值得注意的是，与肾脏自我评估计划相比，肾脏病自我评估计划的准确率更高（58%对 44%；P<0.001）。所有试题的总一致性率为 78%，正确答案的一致性率（84%）高于错误答案（73%）（P<0.001）。在检查各个肾脏病学亚领域时，电解质和酸碱紊乱、肾小球疾病和与肾脏相关的骨骼和结石疾病的总准确率相对较低。GPT-4 回答的总准确率为 74%，高于 GPT-3.5（P<0.001），但仍低于及格线和肾脏病学考生的平均分（77%）。
ChatGPT 在回答与肾脏病学相关的问题时，准确性和可重复性存在局限性。其在不同领域的表现存在差异。
Zotero 插件