Department of Otolaryngology-Head and Neck Surgery, Johns Hopkins University, Baltimore, Maryland, USA.
Center for Surgical Training and Research, Department of Surgery, Michigan Medicine, Ann Arbor, Michigan, USA.
Laryngoscope. 2023 Dec;133(12):3341-3345. doi: 10.1002/lary.30680. Epub 2023 Mar 29.
Competency-based surgical education requires practical assessments and meaningful benchmarks. In otolaryngology, key indicator procedure (KIP) minima are indicators of surgical exposure during training, yet it remains unknown how many times trainees must be evaluated on KIPs to ensure operative competence. Herein, we used Bayesian mixed effects models to compute predicted performance expectations for KIPs.
From November 2017 to September 2021, a smartphone application (SIMPL OR) was used by attendings at five otolaryngology training programs to rate resident operative performance after each case on a five-level scale. Bayesian mixed effects models were used to estimate the probability that postgraduate year (PGY) 3, 4, or 5 trainees would earn a "practice-ready" (PR) rating on a subsequent evaluation based on their previously earned PR ratings for each KIP. Probabilities of earning a subsequent PR rating were examined for interpretability, and cross-validation was used to assess predictive validity.
A total of 842 assessments of KIPs were submitted by 72 attendings for 92 residents PGY 2-5. The predictive model had an average Area Under the Receiver Operating Curve of 0.77. The number of prior PR ratings that senior residents needed to attain a 95% probability of earning a PR rating on a subsequent evaluation was estimated for each KIP. For example, for mastoidectomies, PGY4 residents needed to earn 10 PR ratings whereas PGY5 residents needed 4 PR ratings on average to have a 95% probability of attaining a PR rating on a subsequent evaluation.
Predictive modeling can inform assessment benchmarks for competency-based surgical education.
NA Laryngoscope, 133:3341-3345, 2023.
基于能力的外科教育需要进行实际评估和有意义的基准测试。在耳鼻喉科，关键指标手术（KIP）最低标准是培训期间手术暴露的指标，但仍不清楚学员需要多少次评估 KIP 才能确保手术能力。在此，我们使用贝叶斯混合效应模型来计算 KIP 的预测表现预期。
从 2017 年 11 月至 2021 年 9 月，五家耳鼻喉科培训计划的主治医生使用智能手机应用程序（SIMPL OR）在每次手术后对住院医师的手术表现进行五级评分。使用贝叶斯混合效应模型来估计 PGY3、4 或 5 学员在随后的评估中获得“实践准备”（PR）评级的概率，这是基于他们之前获得的每个 KIP 的 PR 评级。检查了获得后续 PR 评级的概率的可解释性，并使用交叉验证来评估预测有效性。
共有 72 名主治医生为 92 名 PGY2-5 住院医师提交了 842 项 KIP 评估。预测模型的平均接受者操作特征曲线下面积为 0.77。对于每个 KIP，估计了高年级住院医师需要获得多少个 PR 评级才能在随后的评估中获得 95%的 PR 评级的概率。例如，对于乳突切除术，PGY4 住院医师需要获得 10 个 PR 评级，而 PGY5 住院医师平均需要获得 4 个 PR 评级才有 95%的概率在随后的评估中获得 PR 评级。
预测模型可以为基于能力的外科教育提供评估基准。
无 喉镜，133：3341-3345，2023 年。
