Pitts John, Coles Colin, Thomas Peter
NHSE (Wessex Deanery) and King Alfred's College, Winchester, UK.
Med Teach. 2001 Jul;23(4):351-356. doi: 10.1080/01421590120057021.
This paper reports a follow-on project that assessed a series of portfolios assembled by a cohort of participants attending a course for prospective general practice trainers. In an attempt to enhance reliability, a framework for defining and addressing problems using a reflective practice model was offered to participants. The reliability of the judgements made by a panel of assessors about individual 'components', together with an overall global judgement about performance were studied. The reliability of individual assessors' judgements (i.e. their consistency) was moderate, but inter-rater reliability did not reach a level that could support making a safe summative judgement. Despite offering a possible structure for demonstrating reflective processes, the levels of reliability reached were similar to the earlier work and other subjective assessments generally, and perhaps reflected individuality of personal agendas of both the assessed and the assessors, and variations in portfolio structure and content; even agreement among the assessors about evidence of the framework being used was poor. Suggestions for approaches in the future are made. The conclusion remains that while portfolios might be valuable as resources for learning, as assessment tools they should be treated as problematic.
本文报告了一个后续项目，该项目评估了一组参加未来全科实践培训师课程的参与者所组建的一系列档案袋。为提高可靠性，向参与者提供了一个使用反思性实践模型来定义和解决问题的框架。研究了评估小组对各个“组成部分”的判断可靠性，以及对表现的整体综合判断。个体评估者判断的可靠性（即他们的一致性）适中，但评分者间信度未达到能够支持做出可靠总结性判断的水平。尽管提供了一种展示反思过程的可能结构，但所达到的可靠性水平与早期工作以及一般其他主观评估相似，这可能反映了被评估者和评估者个人议程的个性，以及档案袋结构和内容的差异；甚至评估者之间对于使用该框架的证据的一致性也很差。文中提出了对未来方法的建议。结论仍然是，虽然档案袋作为学习资源可能有价值，但作为评估工具，它们应被视为有问题的。
