Israel Hayley P, Slade Martin, Gielissen Katherine, Liu Rachel B, Pisani Margaret A, Chichra Astha
Pulmonary and Critical Care, University of New Mexico School of Medicine, Albuquerque, New Mexico.
Occupational and Environmental Medicine.
ATS Sch. 2023 Jan 24;4(1):61-75. doi: 10.34197/ats-scholar.2022-0063OC. eCollection 2023 Mar.
Existing assessment tools for competence in critical care ultrasound (CCUS) have limited scope and interrupt clinical workflow. The framework of entrustable professional activities (EPAs) is well suited to developing an assessment tool that is comprehensive and readily integrated into the intensive care unit (ICU) training environment.
This study sought to design an EPA-based tool to assess competence in CCUS for pulmonary and critical care fellows and to assess the validity and reliability of the tool.
Eight experts in CCUS met to define the core EPAs for CCUS. A nominal group technique was used to reach consensus. An assessment tool was created based on the EPAs with a modified Ottawa entrustability scale. Trained faculty evaluated pulmonary and critical care fellows using this tool in the ICU over a 6-month study period at a single institution. An assessment of validity of the EPA-based tool is made with four sources of validity evidence: content, response process, reliability, and relation to other variables. Reliability and response process data were generated using generalizability theory analysis to estimate sources of variance in entrustment scores. Analysis of response process validity and validity by relation to other variables was performed using regression models.
Fifty-four assessments were recorded during the study period, conducted on 23 trainees by 13 faculty. Content validity of the tool was demonstrated using expert consensus and published guidelines from critical care societies to define the EPAs. Response process validity was demonstrated by the low variance in entrustment scores due to evaluators (0.086 or 6%) and high agreement between score and trainee self-assessment (regression coefficient, 0.82; < 0.0001). Reliability was demonstrated by the high "true" variance in entrustment score attributable to the trainee: 0.674 or 45%. Validity by relation to other variables was demonstrated using regression analysis to show correlation between entrustment score and the number of times a fellow has performed an EPA (regression coefficient, 0.023; < 0.0001).
An EPA-based assessment tool for competence in CCUS was created. We obtained sufficient validity evidence on three of the diagnostic EPAs. Procedural EPAs were infrequently assessed, limiting generalizability in this subgroup.
现有的重症超声（CCUS）能力评估工具范围有限，且会干扰临床工作流程。可托付专业活动（EPA）框架非常适合开发一种全面且易于融入重症监护病房（ICU）培训环境的评估工具。
本研究旨在设计一种基于EPA的工具，以评估肺科和重症监护专科培训学员的CCUS能力，并评估该工具的有效性和可靠性。
八位CCUS专家会面，确定CCUS的核心EPA。采用名义小组技术达成共识。基于EPA并使用改良的渥太华可托付性量表创建了一个评估工具。在单一机构的6个月研究期间，经过培训的教员在ICU使用该工具对肺科和重症监护专科培训学员进行评估。基于EPA的工具的有效性评估采用四种有效性证据来源：内容、反应过程、可靠性以及与其他变量的关系。可靠性和反应过程数据通过概化理论分析生成，以估计托付分数的方差来源。使用回归模型进行反应过程有效性和与其他变量相关的有效性分析。
在研究期间记录了54次评估，由13名教员对23名学员进行。通过专家共识和重症监护学会发布的指南来定义EPA，证明了该工具的内容有效性。由于评估者导致的托付分数方差较低（0.086或6%），且分数与学员自我评估之间的一致性较高（回归系数，0.82；<0.0001），证明了反应过程有效性。托付分数中归因于学员的高“真实”方差：0.674或45%，证明了可靠性。使用回归分析显示托付分数与学员执行EPA的次数之间的相关性（回归系数，0.023；<0.0001），证明了与其他变量相关的有效性。
创建了一种基于EPA的CCUS能力评估工具。我们在三个诊断性EPA上获得了足够的有效性证据。程序性EPA评估较少，限制了该亚组的普遍性。
