From the Department of Anesthesiology, Perioperative and Pain Medicine, Boston Children's Hospital and Harvard Medical School, Boston, Massachusetts (R.H.B.); the Department of Surgery, Emory University School of Medicine, Atlanta, Georgia (S.L.M.-W.); the Foundation for Advancement of International Medical Education and Research, Philadelphia, Pennsylvania (J.R.B.); the Center for Medical Simulation, Charlestown, Massachusetts (J.B.C., R.H.B.); and the Department of Anesthesia, Critical Care and Pain Medicine (J.B.C.), the Department of Surgery and Massachusetts General Hospital Learning Laboratory (E.R.P.), Massachusetts General Hospital and Harvard Medical School, Boston, Massachusetts. Massachusetts General Hospital, Harvard Medical School, Boston, Massachusetts Brigham and Women's Hospital, Harvard Medical School, Boston, Massachusetts Children's Hospital Boston, Harvard Medical School, Boston, Massachusetts Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, Massachusetts Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, Massachusetts Massachusetts General Hospital, Harvard Medical School, Boston, Massachusetts Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, Massachusetts Center for Medical Simulation, Massachusetts General Hospital, Harvard Medical School, Boston, Massachusetts Newton-Wellesley Hospital, Newton, Massachusetts Massachusetts General Hospital, Harvard Medical School, Boston, Massachusetts Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, Massachusetts Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, Massachusetts Brigham and Women's Hospital, Harvard Medical School, Boston, Massachusetts Brigham and Women's Hospital, Harvard Medical School, Boston, Massachusetts Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, Massachusetts Brigham and Women's Hospital, Harvard Medical School, Boston, Massachusetts Brigham and Women's Hospital, Harvard Medical School, Boston, Massachusetts Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, Massachusetts Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, Massachusetts Brigham and Women's Hospital, Harvard Medical School, Boston, Massachusetts Massachusetts General Hospital, Harvard Medical School, Boston, Massachusetts Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, Massachusetts Massachusetts General Hospital, Harvard Medical School, Boston, Massachusetts Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, Massachusetts Brigham and Women's Hospital, Harvard Medical School, Boston, Massachusetts Brigham and Women's Hospital, Harvard Medical School, Boston, Massachusetts Brigham and Women's Hospital, Harvard Medical School, Boston, Massachusetts Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, Massachusetts Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, Massachusetts.
Anesthesiology. 2018 Apr;128(4):821-831. doi: 10.1097/ALN.0000000000002091.
Obtaining reliable and valid information on resident performance is critical to patient safety and training program improvement. The goals were to characterize important anesthesia resident performance gaps that are not typically evaluated, and to further validate scores from a multiscenario simulation-based assessment.
Seven high-fidelity scenarios reflecting core anesthesiology skills were administered to 51 first-year residents (CA-1s) and 16 third-year residents (CA-3s) from three residency programs. Twenty trained attending anesthesiologists rated resident performances using a seven-point behaviorally anchored rating scale for five domains: (1) formulate a clear plan, (2) modify the plan under changing conditions, (3) communicate effectively, (4) identify performance improvement opportunities, and (5) recognize limits. A second rater assessed 10% of encounters. Scores and variances for each domain, each scenario, and the total were compared. Low domain ratings (1, 2) were examined in detail.
Interrater agreement was 0.76; reliability of the seven-scenario assessment was r = 0.70. CA-3s had a significantly higher average total score (4.9 ± 1.1 vs. 4.6 ± 1.1, P = 0.01, effect size = 0.33). CA-3s significantly outscored CA-1s for five of seven scenarios and domains 1, 2, and 3. CA-1s had a significantly higher proportion of worrisome ratings than CA-3s (chi-square = 24.1, P < 0.01, effect size = 1.50). Ninety-eight percent of residents rated the simulations more educational than an average day in the operating room.
Sensitivity of the assessment to CA-1 versus CA-3 performance differences for most scenarios and domains supports validity. No differences, by experience level, were detected for two domains associated with reflective practice. Smaller score variances for CA-3s likely reflect a training effect; however, worrisome performance scores for both CA-1s and CA-3s suggest room for improvement.
获取关于住院医师表现的可靠和有效的信息对患者安全和培训计划的改进至关重要。本研究的目的是描述通常未评估的重要麻醉住院医师表现差距，并进一步验证基于多情景模拟评估的评分。
从三个住院医师培训项目中，向 51 名一年级住院医师（CA-1）和 16 名三年级住院医师（CA-3）施行了七个反映核心麻醉学技能的高保真情景。20 名经过培训的主治麻醉师使用七点评分制行为锚定评分量表对住院医师的表现进行评估，该量表用于五个领域：（1）制定明确的计划，（2）根据变化的情况修改计划，（3）进行有效的沟通，（4）识别绩效改进机会，以及（5）认识到局限性。第二位评估员评估了 10%的情况。比较了每个领域、每个情景和总分的分数和方差。详细检查了低领域评分（1、2）。
评分者间的一致性为 0.76；七个情景评估的可靠性为 r = 0.70。CA-3 的平均总分（4.9±1.1 分比 4.6±1.1 分，P = 0.01，效应大小 = 0.33）显著较高。CA-3 在七个情景和三个领域中的五个情景中得分均显著高于 CA-1。CA-1 在五个情景和两个领域中的表现均显著低于 CA-3。CA-1 的担忧评分比例显著高于 CA-3（卡方 = 24.1，P < 0.01，效应大小 = 1.50）。98%的住院医师认为模拟比手术室的平均一天更具教育意义。
评估对大多数情景和领域中 CA-1 与 CA-3 表现差异的敏感性支持其有效性。在与反思实践相关的两个领域，没有发现按经验水平的差异。CA-3 的分数方差较小可能反映了培训效果；然而，CA-1 和 CA-3 的表现都存在令人担忧的评分，这表明仍有改进的空间。
