Tor Elina, Steketee Carole, Mak Donna
School of Medicine Fremantle, The University of Notre Dame Australia, Fremantle, Western Australia, Australia.
Learning and Teaching Office, The University of Notre Dame Australia, Fremantle, Western Australia, Australia.
Int J Med Educ. 2016 Sep 24;7:309-319. doi: 10.5116/ijme.57da.c89a.
To evaluate the merit of the Clinical Audit Project (CAP) in an assessment program for undergraduate medical education using a systematic assessment validation framework.
A cross-sectional assessment validation study at one medical school in Western Australia, with retrospective qualitative analysis of the design, development, implementation and outcomes of the CAP, and quantitative analysis of assessment data from four cohorts of medical students (2011- 2014).
The CAP is fit for purpose with clear external and internal alignment to expected medical graduate outcomes. Substantive validity in students' and examiners' response processes is ensured through relevant methodological and cognitive processes. Multiple validity features are built-in to the design, planning and implementation process of the CAP. There is evidence of high internal consistency reliability of CAP scores (Cronbach's alpha > 0.8) and inter-examiner consistency reliability (intra-class correlation>0.7). Aggregation of CAP scores is psychometrically sound, with high internal consistency indicating one common underlying construct. Significant but moderate correlations between CAP scores and scores from other assessment modalities indicate validity of extrapolation and alignment between the CAP and the overall target outcomes of medical graduates. Standard setting, score equating and fair decision rules justify consequential validity of CAP scores interpretation and use.
This study provides evidence demonstrating that the CAP is a meaningful and valid component in the assessment program. This systematic framework of validation can be adopted for all levels of assessment in medical education, from individual assessment modality, to the validation of an assessment program as a whole.
使用系统的评估验证框架，评估临床审计项目（CAP）在本科医学教育评估项目中的价值。
在西澳大利亚的一所医学院开展横断面评估验证研究，对CAP的设计、开发、实施和结果进行回顾性定性分析，并对四批医学生（2011 - 2014年）的评估数据进行定量分析。
CAP符合预期目标，在外部和内部与预期的医学毕业生成果明确一致。通过相关的方法和认知过程，确保了学生和考官反应过程中的实质有效性。CAP的设计、规划和实施过程内置了多种有效性特征。有证据表明CAP分数具有较高的内部一致性信度（Cronbach's alpha > 0.8）和考官间一致性信度（组内相关系数>0.7）。CAP分数的汇总在心理测量学上是合理的，高内部一致性表明存在一个共同的潜在结构。CAP分数与其他评估方式的分数之间存在显著但适度的相关性，这表明CAP与医学毕业生的总体目标成果之间具有外推和一致性的有效性。标准设定、分数等值和公平决策规则证明了CAP分数解释和使用的结果有效性。
本研究提供的证据表明，CAP是评估项目中有意义且有效的组成部分。这种系统的验证框架可应用于医学教育的各级评估，从单个评估方式到整个评估项目的验证。
