Wake Forest University School of Medicine, Winston-Salem, USA.
University of Minnesota School of Medicine, Minneapolis, USA.
Adv Health Sci Educ Theory Pract. 2024 May;29(2):567-585. doi: 10.1007/s10459-023-10269-0. Epub 2023 Aug 2.
Education in Doctor of Medicine programs has moved towards an emphasis on clinical competency, with entrustable professional activities providing a framework of learning objectives and outcomes to be assessed within the clinical environment. While the identification and structured definition of objectives and outcomes have evolved, many methods employed to assess clerkship students' clinical skills remain relatively unchanged. There is a paucity of medical education research applying advanced statistical design and analytic techniques to investigate the validity of clinical skills assessment. One robust statistical method, multitrait-multimethod matrix analysis, can be applied to investigate construct validity across multiple assessment instruments and settings. Four traits were operationalized to represent the construct of critical clinical skills (professionalism, data gathering, data synthesis, and data delivery). The traits were assessed using three methods (direct observations by faculty coaches, clinical workplace-based evaluations, and objective structured clinical examination type clinical practice examinations). The four traits and three methods were intercorrelated for the multitrait-multimethod matrix analysis. The results indicated reliability values in the adequate to good range across the three methods with the majority of the validity coefficients demonstrating statistical significance. The clearest evidence for convergent and divergent validity was with the professionalism trait. The correlations on the same method/different traits analyses indicated substantial method effect; particularly on clinical workplace-based assessments. The multitrait-multimethod matrix approach, currently underutilized in medical education, could be employed to explore validity evidence of complex constructs such as clinical skills. These results can inform faculty development programs to improve the reliability and validity of assessments within the clinical environment.
医学博士课程的教育已转向强调临床能力，可信赖的专业活动为临床环境中的学习目标和成果评估提供了框架。虽然目标和成果的识别和结构化定义已经发展，但评估实习学生临床技能的许多方法仍然相对不变。应用高级统计设计和分析技术来研究临床技能评估的有效性的医学教育研究很少。一种强大的统计方法，多特质-多方法矩阵分析，可以应用于研究多个评估工具和环境中的构念效度。四个特质被操作化以代表关键临床技能的结构（专业精神、数据收集、数据综合和数据传递）。这些特质使用三种方法进行评估（教师教练的直接观察、临床基于工作场所的评估和客观结构化临床考试类型的临床实践考试）。多特质-多方法矩阵分析对四个特质和三种方法进行了相互关联。结果表明，在三种方法中，可靠性值在足够到良好的范围内，大多数有效性系数具有统计学意义。在专业精神特质方面，有最明显的收敛和发散有效性证据。在相同方法/不同特质分析上的相关性表明存在实质性的方法效应；特别是在临床基于工作场所的评估上。在医学教育中尚未充分利用的多特质-多方法矩阵方法，可以用于探索临床技能等复杂结构的有效性证据。这些结果可以为教师发展计划提供信息，以提高临床环境中评估的可靠性和有效性。
