Lehigh Valley Health Network, Department of Emergency and Hospital Medicine, Bethlehem, Pennsylvania.
University of South Florida Morsani College of Medicine, Tampa, Florida.
West J Emerg Med. 2019 Jan;20(1):64-70. doi: 10.5811/westjem.2018.12.39844. Epub 2018 Dec 17.
The Accreditation Council for Graduate Medical Education (ACGME) specifically notes multisource feedback (MSF) as a recommended means of resident assessment in the emergency medicine (EM) Milestones. High-fidelity simulation is an environment wherein residents can receive MSF from various types of healthcare professionals. Previously, the Queen's Simulation Assessment Tool (QSAT) has been validated for faculty to assess residents in five categories: assessment; diagnostic actions; therapeutic actions; interpersonal communication, and overall assessment. We sought to determine whether the QSAT could be used to provide MSF using a standardized simulation case.
Prospectively after institutional review board approval, residents from a dual ACGME/osteopathic-approved postgraduate years (PGY) 1-4 EM residency were consented for participation. We developed a standardized resuscitation after overdose case with specific 1-5 Likert anchors used by the QSAT. A PGY 2-4 resident participated in the role of team leader, who completed a QSAT as self-assessment. The team consisted of a PGY-1 peer, an emergency medical services (EMS) provider, and a nurse. Two core faculty were present to administer the simulation case and assess. Demographics were gathered from all participants completing QSATs. We analyzed QSATs by each category and on cumulative score. Hypothesis testing was performed using intraclass correlation coefficients (ICC), with 95% confidence intervals. Interpretation of ICC results was based on previously published definitions.
We enrolled 34 team leader residents along with 34 nurses. A single PGY-1, a single EMS provider and two faculty were also enrolled. Faculty provided higher cumulative QSAT scores than the other sources of MSF. QSAT scores did not increase with team leader PGY level. ICC for inter-rater reliability for all sources of MSF was 0.754 (0.572-0.867). Removing the self-evaluation scores increased inter-rater reliability to 0.838 (0.733-0.910). There was lesser agreement between faculty and nurse evaluations than from the EMS or peer evaluation.
In this single-site cohort using an internally developed simulation case, the QSAT provided MSF with excellent reliability. Self-assessment decreases the reliability of the MSF, and our data suggest self-assessment should not be a component of MSF. Use of the QSAT for MSF may be considered as a source of data for clinical competency committees.
研究生医学教育认证委员会（ACGME）特别指出，多源反馈（MSF）是急诊医学（EM）里程碑式评估住院医师的推荐方法。高保真模拟是住院医师可以从各种类型的医疗保健专业人员那里获得 MSF 的环境。此前，皇后模拟评估工具（QSAT）已通过验证，可用于教职员工评估五个类别中的住院医师：评估；诊断行动；治疗行动；人际沟通和总体评估。我们试图确定 QSAT 是否可以用于使用标准化模拟案例提供 MSF。
在机构审查委员会批准后，从双重 ACGME/骨科认证的住院医师 1-4 年 EM 住院医师中前瞻性地同意参与。我们开发了一种标准化的过量后复苏案例，使用 QSAT 的特定 1-5 级李克特锚定。一名住院医师 2-4 年的住院医师担任团队负责人，完成了自我评估的 QSAT。团队由一名住院医师 1 年的同伴、一名紧急医疗服务（EMS）提供者和一名护士组成。两位核心教员出席以管理模拟案例并进行评估。从所有完成 QSAT 的参与者中收集人口统计学信息。我们按每个类别和累积分数分析 QSAT。使用组内相关系数（ICC）进行假设检验，并具有 95%置信区间。ICC 结果的解释基于先前发表的定义。
我们招募了 34 名团队负责人住院医师，以及 34 名护士。还招募了一名住院医师 1 年、一名 EMS 提供者和两名教员。教员提供的累积 QSAT 分数高于其他 MSF 来源。QSAT 分数并未随团队负责人的 PGY 水平而增加。所有 MSF 来源的内部一致性信度 ICC 为 0.754（0.572-0.867）。删除自我评估分数后，内部一致性信度增加到 0.838（0.733-0.910）。教员和护士的评估之间的一致性小于 EMS 或同伴评估。
在这项使用内部开发的模拟案例的单站点队列研究中，QSAT 提供了具有出色可靠性的 MSF。自我评估降低了 MSF 的可靠性，我们的数据表明自我评估不应成为 MSF 的组成部分。QSAT 可用于 MSF 可能被视为临床能力委员会数据的来源。
