Department of Educational Development and Research, Maastricht University, Universiteitssingel 60, 6229 ER, Maastricht, The Netherlands.
Center for Education Development and Research in Health Professions (CEDAR), Faculty of Medical Sciences, University Medical Center Groningen, University of Groningen, Groningen, The Netherlands.
Adv Health Sci Educ Theory Pract. 2018 May;23(2):275-287. doi: 10.1007/s10459-017-9793-y. Epub 2017 Oct 14.
While portfolios are increasingly used to assess competence, the validity of such portfolio-based assessments has hitherto remained unconfirmed. The purpose of the present research is therefore to further our understanding of how assessors form judgments when interpreting the complex data included in a competency-based portfolio. Eighteen assessors appraised one of three competency-based mock portfolios while thinking aloud, before taking part in semi-structured interviews. A thematic analysis of the think-aloud protocols and interviews revealed that assessors reached judgments through a 3-phase cyclical cognitive process of acquiring, organizing, and integrating evidence. Upon conclusion of the first cycle, assessors reviewed the remaining portfolio evidence to look for confirming or disconfirming evidence. Assessors were inclined to stick to their initial judgments even when confronted with seemingly disconfirming evidence. Although assessors reached similar final (pass-fail) judgments of students' professional competence, they differed in their information-processing approaches and the reasoning behind their judgments. Differences sprung from assessors' divergent assessment beliefs, performance theories, and inferences about the student. Assessment beliefs refer to assessors' opinions about what kind of evidence gives the most valuable and trustworthy information about the student's competence, whereas assessors' performance theories concern their conceptualizations of what constitutes professional competence and competent performance. Even when using the same pieces of information, assessors furthermore differed with respect to inferences about the student as a person as well as a (future) professional. Our findings support the notion that assessors' reasoning in judgment and decision-making varies and is guided by their mental models of performance assessment, potentially impacting feedback and the credibility of decisions. Our findings also lend further credence to the assertion that portfolios should be judged by multiple assessors who should, moreover, thoroughly substantiate their judgments. Finally, it is suggested that portfolios be designed in such a way that they facilitate the selection of and navigation through the portfolio evidence.
虽然组合越来越多地被用于评估能力，但此类基于组合的评估的有效性迄今仍未得到证实。因此，本研究的目的是进一步了解评估者在解释基于能力的组合中包含的复杂数据时如何形成判断。18 名评估者在进行半结构化访谈之前，通过出声思考的方式对三个基于能力的模拟组合中的一个进行评估。对出声思考记录和访谈的主题分析表明，评估者通过获取、组织和整合证据的三阶段循环认知过程做出判断。在第一周期结束时，评估者会审查剩余的组合证据，以寻找确认或否定的证据。即使面对看似否定的证据，评估者也倾向于坚持最初的判断。尽管评估者对学生的专业能力做出了相似的最终（通过/失败）判断，但他们在信息处理方法和判断背后的推理上存在差异。差异源于评估者不同的评估信念、绩效理论以及对学生的推断。评估信念是指评估者对什么样的证据能提供最有价值和最可信的关于学生能力的信息的看法，而评估者的绩效理论则涉及他们对专业能力和胜任表现的概念化。即使使用相同的信息，评估者在关于学生作为个人以及（未来）专业人员的推断上也存在差异。我们的研究结果支持这样一种观点，即评估者在判断和决策中的推理是不同的，并且受到他们对绩效评估的心理模型的指导，这可能会影响反馈和决策的可信度。我们的研究结果还进一步证实了这样一种说法，即组合应由多名评估者进行评估，而且评估者应该详细说明他们的判断依据。最后，建议设计组合的方式应便于选择和浏览组合证据。
