Najafali Daniel, Galbraith Logan G, Camacho Justin M, Stoffel Victoria, Herzog Isabel, Moss Civanni, Taiberg Stephanie L, Knoedler Leonard
From the Carle Illinois College of Medicine, University of Illinois Urbana-Champaign, Urbana, Ill.
Northeast Ohio Medical University College of Medicine, Rootstown, Ohio.
Plast Reconstr Surg Glob Open. 2024 Sep 19;12(9):e6185. doi: 10.1097/GOX.0000000000006185. eCollection 2024 Sep.
The Plastic Surgery In-Service Training Examination (PSITE) remains a critical milestone in residency training. Successful preparation requires extensive studying during an individual's residency. This study focuses on the capacity of Generative Pre-trained Transformer 4 (GPT-4) to generate PSITE practice questions.
GPT-4 was prompted to generate multiple choice questions for each PSITE section and provide answer choices with detailed rationale. Question composition via readability metrics were analyzed, along with quality. Descriptive statistics compared GPT-4 and the 2022 PSITE.
The overall median Flesch-Kincaid reading ease for GPT-4-generated questions was 43.90 (versus 50.35 PSITE, = 0.036). GPT-4 provided questions that contained significantly fewer mean sentences (1 versus 4), words (16 versus 56), and percentage of complex words (3 versus 13) than 2022 PSITE questions ( < 0.001). When evaluating GPT-4 generated questions for each examination section, the highest median Flesch-Kincaid reading ease was on the core surgical principles section (median: 63.30, interquartile range [54.45-68.28]) and the lowest was on the craniomaxillofacial section (median: 36.25, interquartile range [12.57-58.40]). Most readability metrics were higher for the 2022 PSITE compared with GPT-4 generated questions. Overall question quality was poor for the chatbot.
Our study found that GPT-4 can be adapted to generate practice questions for the 2022 PSITE, but its questions are of poor quality. The program can offer general explanations for both the correct and incorrect answer options but was observed to generate false information and poor-quality explanations. Although trainees should navigate with caution as the technology develops, GPT-4 has the potential to serve as an effective educational adjunct under the supervision of trained plastic surgeons.
整形外科学术在职培训考试（PSITE）仍是住院医师培训中的一个关键里程碑。成功备考需要在个人住院医师培训期间进行广泛学习。本研究聚焦于生成式预训练变换器4（GPT-4）生成PSITE练习题的能力。
促使GPT-4为每个PSITE部分生成多项选择题，并提供带有详细理由的答案选项。通过可读性指标对问题构成进行分析，并评估质量。采用描述性统计对GPT-4和2022年的PSITE进行比较。
GPT-4生成问题的总体弗莱什-金凯德易读性中位数为43.90（相比之下，PSITE为50.35，P = 0.036）。与2022年PSITE问题相比，GPT-4提供的问题平均句子数（1个对4个）、单词数（16个对56个）和复杂单词百分比（3%对13%）显著更少（P < 0.001）。在评估GPT-4为每个考试部分生成的问题时，弗莱什-金凯德易读性中位数最高的是核心外科原则部分（中位数：63.30，四分位间距[54.45 - 68.28]），最低的是颅颌面部分（中位数：36.25，四分位间距[12.57 - 58.40]）。与GPT-4生成的问题相比，2022年PSITE的大多数可读性指标更高。聊天机器人生成问题的总体质量较差。
我们的研究发现，GPT-4可被改编用于生成2022年PSITE的练习题，但其问题质量较差。该程序可为正确和错误答案选项提供一般性解释，但被观察到会生成错误信息和质量不佳的解释。尽管随着技术发展学员应谨慎使用，但GPT-4有潜力在训练有素的整形外科医生监督下作为一种有效的教育辅助工具。
