Sinz Elizabeth, Banerjee Arna, Steadman Randolph, Shotwell Matthew S, Slagle Jason, McIvor William R, Torsher Laurence, Burden Amanda, Cooper Jeffrey B, DeMaria Samuel, Levine Adam I, Park Christine, Gaba David M, Weinger Matthew B, Boulet John R
Penn State University College of Medicine, Hershey, PA, 17033, USA.
Vanderbilt University School of Medicine, Nashville, TN, USA.
BMC Med Educ. 2021 Apr 12;21(1):207. doi: 10.1186/s12909-021-02617-8.
Even physicians who routinely work in complex, dynamic practices may be unprepared to optimally manage challenging critical events. High-fidelity simulation can realistically mimic critical clinically relevant events, however the reliability and validity of simulation-based assessment scores for practicing physicians has not been established.
Standardised complex simulation scenarios were developed and administered to board-certified, practicing anesthesiologists who volunteered to participate in an assessment study during formative maintenance of certification activities. A subset of the study population agreed to participate as the primary responder in a second scenario for this study. The physicians were assessed independently by trained raters on both teamwork/behavioural and technical performance measures. Analysis using Generalisability and Decision studies were completed for the two scenarios with two raters.
The behavioural score was not more reliable than the technical score. With two raters > 20 scenarios would be required to achieve a reliability estimate of 0.7. Increasing the number of raters for a given scenario would have little effect on reliability.
The performance of practicing physicians on simulated critical events may be highly context-specific. Realistic simulation-based assessment for practicing physicians is resource-intensive and may be best-suited for individualized formative feedback. More importantly, aggregate data from a population of participants may have an even higher impact if used to identify skill or knowledge gaps to be addressed by training programs and inform continuing education improvements across the profession.
即使是那些日常工作环境复杂多变的医生，在应对具有挑战性的危急事件时也可能准备不足。高保真模拟能够逼真地模拟关键的临床相关事件，然而，针对执业医生基于模拟的评估分数的可靠性和有效性尚未得到确立。
开发了标准化的复杂模拟场景，并应用于那些自愿参与认证维持阶段形成性评估研究的获得委员会认证的执业麻醉医师。研究人群中的一部分同意作为主要应答者参与本研究的第二个场景。由经过培训的评估者对医生在团队协作/行为和技术表现方面进行独立评估。针对两个场景，由两名评估者完成了概化性分析和决策分析。
行为评分的可靠性并不高于技术评分。若有两名评估者，要达到0.7的可靠性估计需要超过20个场景。对于给定场景增加评估者数量对可靠性影响不大。
执业医生在模拟危急事件中的表现可能高度依赖具体情境。基于现实模拟对执业医生进行评估资源消耗大，可能最适合提供个性化的形成性反馈。更重要的是，如果将来自一群参与者的汇总数据用于识别培训项目需要解决的技能或知识差距，并为整个行业的继续教育改进提供参考，可能会产生更大的影响。
