Department of Tertiary Education (Assessment), Australian Council for Educational Research, Camberwell, Australia.
Department of Medical Education, University of Melbourne, Australia & Educational Monitoring and Research, Australian Council for Educational Research, Camberwell, Australia.
Med Teach. 2021 May;43(5):567-574. doi: 10.1080/0142159X.2021.1878122. Epub 2021 Feb 8.
A programmatic approach to assessment entails gathering and aggregating 'rich information' on candidates to inform progress decisions. However, there is little guidance on how such an approach might be implemented in practice.
We describe an approach to aggregating rich information across assessment formats to inform committee decision-making in a specialist medical college.
Each item (n = 272) for every examination was blueprinted to 15 curriculum modules and 7 proficiencies. We developed a six-point holistic rating scale with detailed rubrics outlining expected performance standards for every item. Examiners used this rating scale in making judgements for each item, generating rich performance data for each candidate.
A colour-coded 'mosaic' of patterns of performance across modules and proficiencies was generated along with frequency distributions of ratings. These data allowed examiners to easily visualise candidate performance and to use these data to inform deliberations on borderline candidates. Committee decision-making was facilitated by maintaining the richness of assessment information throughout the process. Moreover, the data facilitated detailed and useful feedback to candidates.
Our study demonstrates that incorporating aspects of programmatic thinking into high-stakes examinations by using a novel approach to aggregating information is a useful first step in reforming an assessment program.
评估的计划性方法需要收集和汇总候选人的“丰富信息”，以告知进展决策。然而，关于如何在实践中实施这种方法，几乎没有指导。
我们描述了一种在专家医学院委员会决策中汇总评估格式的丰富信息的方法。
每个考试的每个项目（n=272）都按照 15 个课程模块和 7 个能力进行蓝图设计。我们制定了一个六分制整体评分量表，详细的评分标准概述了每个项目的预期表现标准。考官使用此评分量表对每个项目进行判断，为每个候选人生成丰富的表现数据。
生成了模块和能力的表现模式的彩色镶嵌图，以及评分的频率分布。这些数据使考官能够轻松地可视化候选人的表现，并使用这些数据为边缘候选人的审议提供信息。通过在整个过程中保持评估信息的丰富性，委员会决策得到了促进。此外，数据为候选人提供了详细而有用的反馈。
我们的研究表明，通过使用新颖的信息汇总方法将计划性思维的各个方面纳入高风险考试中，是改革评估计划的有用的第一步。
