O'Keeffe Dara A, Traynor Oscar, Tekian Ara, Park Yoon Soo
National Surgical and Clinical Skills Centre, Royal College of Surgeons in Ireland University of Medicine and Health Sciences, Dublin, Ireland.
National Surgical and Clinical Skills Centre, Royal College of Surgeons in Ireland University of Medicine and Health Sciences, Dublin, Ireland.
J Surg Educ. 2024 Nov;81(11):1709-1719. doi: 10.1016/j.jsurg.2024.08.008. Epub 2024 Sep 18.
The objective of this study was to evaluate the validity evidence supporting the use and interpretation of a multifaceted assessment system in the early years of surgical training.
This was a national retrospective cohort study analyzing the validity and reliability of an assessment process for surgical residents over a 2-year period. Data from all elements of the assessment process was evaluated using Messick's unified validity framework. Assessments were categorized as Workplace-based, Structured assessment performed in the academic center and Multiple Mini Interview.
Our Institution is a health sciences university and the body responsible for the training and certification of all surgeons in our national program. Residents on the Core Surgical Training program undergo multiple assessments over the first 2 years of postgraduate training, both in the workplace and the academic training center, which inform their progression into higher surgical training in their chosen specialty.
Data was collected from 2 cohorts of the entire population of postgraduate trainees nationally (N = 114).
Best practice standards for educational testing aligned with the results supporting the use of this assessment process. Findings indicate a robust assessment system, demonstrating validity evidence in content, response process, internal structure, relations to other variables, and consequences. Composite score reliability of the assessment was 0.89 which demonstrates a highly reliable process. Correlation between workplace-based assessments and standardized tests performed in the simulation setting was also very high (0.93).
The Core Training Assessment System (CTAS) provides a psychometrically rigorous system to measure trainee competence during the initial years of training. Residency programs of all sizes can replicate the methods described here to demonstrate the validity of their assessment processes, thereby being able to stand over decisions on surgical competency.
本研究的目的是评估支持在外科培训早期使用和解释多方面评估系统的有效性证据。
这是一项全国性回顾性队列研究，分析了为期两年的外科住院医师评估过程的有效性和可靠性。使用梅西克的统一有效性框架对评估过程中所有要素的数据进行了评估。评估分为基于工作场所的评估、在学术中心进行的结构化评估和多次迷你面试。
我们的机构是一所健康科学大学，是负责我们国家项目中所有外科医生培训和认证的机构。参加核心外科培训项目的住院医师在研究生培训的前两年要在工作场所和学术培训中心接受多次评估，这些评估为他们进入所选专业的更高阶段外科培训提供依据。
从全国研究生培训学员的2个队列中收集了数据（N = 114）。
教育测试的最佳实践标准与支持使用该评估过程的结果一致。研究结果表明该评估系统强大，在内容、反应过程、内部结构、与其他变量的关系以及结果方面都有有效性证据。该评估的综合评分信度为0.89，表明该过程高度可靠。基于工作场所的评估与在模拟环境中进行的标准化测试之间的相关性也非常高（0.93）。
核心培训评估系统（CTAS）提供了一个心理测量严格的系统，用于在培训初期衡量学员的能力。各种规模的住院医师培训项目都可以复制此处描述的方法，以证明其评估过程的有效性，从而能够坚持关于外科能力的决策。
