Monteiro Sandra, McConnell Meghan M
Division of Education and Innovation, Department of Medicine, Faculty of Health Sciences Program in Education Research, Innovation, and Theory (MERIT), Centre for Simulation Based Learning (CSBL), McMaster University, David Braley Health Sciences Centre, 100 Main Street, 5Th Floor 5002 A/E, Hamilton, Canada.
Department of Innovation in Medical Education (DIME), University of Ottawa, Ottawa, Canada.
Med Sci Educ. 2023 May 8;33(3):729-736. doi: 10.1007/s40670-023-01794-z. eCollection 2023 Jun.
A competency-based framework focuses on alignment between professional standards and assessment design. This alignment implies improved measurement validity, yet it has not been established that competence in one context predicts performance in another context. High-stakes competence assessments offer insights into the relationship between assessment design and competencies.
METHODS/ANALYSES: The internationally educated nurses competency assessment program (IENCAP) was developed at Touchstone Institute in collaboration with the College of Nurses of Ontario (CNO) and includes a 12-station OSCE. Each station evaluated the same 10 competencies. We submitted competency scores to a multi-trait multi-method matrix analysis to evaluate the convergent and discriminant validity of competencies.
RESULTS/OBSERVATIONS: All correlations were significant and positive; however, we did not find evidence of convergent or discriminant validity. Correlations were higher between different competencies evaluated within the same station (mean correlation = 0.60) compared to identical competencies evaluated across different stations (mean correlation = 0.19).
The results do not provide evidence of construct validity for competencies. While competency-based approaches emphasize various generalized knowledge, skills, and attitudes, these findings indicate that the clinical context is a major determinant of performance.
The context-dependent nature of competencies requires multiple assessments in varied contexts. Performance on a single competency cannot be determined in a single occasion.
The online version contains supplementary material available at 10.1007/s40670-023-01794-z.
基于能力的框架注重专业标准与评估设计之间的一致性。这种一致性意味着测量效度的提高，但尚未确定在一种情境下的能力能否预测在另一种情境下的表现。高风险能力评估为评估设计与能力之间的关系提供了见解。
方法/分析：国际教育护士能力评估项目（IENCAP）由基石研究所与安大略省护士学院（CNO）合作开发，包括一个有12个站点的客观结构化临床考试（OSCE）。每个站点评估相同的10项能力。我们将能力得分提交给多特质多方法矩阵分析，以评估能力的聚合效度和区分效度。
结果/观察结果：所有相关性均为显著正相关；然而，我们没有发现聚合效度或区分效度的证据。与跨不同站点评估的相同能力相比，同一站点内评估的不同能力之间的相关性更高（平均相关性 = 0.60）（平均相关性 = 0.19）。
结果未提供能力的结构效度证据。虽然基于能力的方法强调各种广义的知识、技能和态度，但这些发现表明临床情境是表现的主要决定因素。
能力的情境依赖性需要在不同情境下进行多次评估。不能在单一情况下确定单一能力的表现。
在线版本包含可在10.1007/s40670-023-01794-z获取的补充材料。
