Department of Medical Education, College of Medicine, The Catholic University of Korea, Seoul, Korea.
J Educ Eval Health Prof. 2020;17:39. doi: 10.3352/jeehp.2020.17.39. Epub 2020 Dec 9.
Consistent evaluation procedures based on objective and rational standards are essential for the sustainability of portfolio-based education, which has been widely introduced in medical education. We aimed to develop and implement a portfolio assessment system, and to assess its validity and reliability.
We developed a portfolio assessment system from March 2019 to August 2019 and confirmed its content validity through expert assessment by an expert group comprising 2 medical education specialists, 2 professors involved in education at medical school, and a professor of basic medical science. Six trained assessors conducted 2 rounds of evaluation of 7 randomly selected portfolios for the “Self-Development and Portfolio II” course from January 2020 to July 2020. These data are used inter-rater reliability was evaluated using intra-class correlation coefficients (ICCs) in September 2020.
The portfolio assessment system is based on the following process; assessor selection, training, analytical/comprehensive evaluation, and consensus. Appropriately trained assessors evaluated portfolios based on specific assessment criteria and a rubric for assigning points. In the analysis of inter-rater reliability, the first round of evaluation grades was submitted, and all assessment areas except “goal-setting” showed a high ICC of 0.81 or higher. After the first round of assessment, we attempted to standardize objective assessment procedures. As a result, all components of the assessments showed close correlations, with ICCs of 0.81 or higher.
We confirmed that when assessors with an appropriate training conduct portfolio assessment based on specified standards through a systematic procedure, the results are reliable.
基于客观和理性标准的一致评估程序对于基于组合的教育的可持续性至关重要，这种教育已经在医学教育中得到广泛应用。我们旨在开发和实施一种组合评估系统，并评估其有效性和可靠性。
我们从 2019 年 3 月至 2019 年 8 月开发了一个组合评估系统，并通过由 2 名医学教育专家、2 名医学院教育教授和 1 名基础医学教授组成的专家组进行专家评估来确认其内容有效性。2020 年 1 月至 2020 年 7 月，6 名经过培训的评估员对“自我发展和组合 II”课程的 7 份随机组合进行了两轮评估。2020 年 9 月，使用组内相关系数（ICC）评估这些数据的组内可靠性。
组合评估系统基于以下过程进行：评估员选择、培训、分析/综合评估和共识。经过适当培训的评估员根据特定的评估标准和评分量表对组合进行评估。在评估者间可靠性的分析中，提交了第一轮评估等级，除“设定目标”外，所有评估领域的 ICC 均为 0.81 或更高。在第一轮评估后，我们试图标准化客观评估程序。结果，所有评估部分的相关性都很高，ICC 为 0.81 或更高。
我们确认，当具有适当培训的评估员通过系统的程序根据规定的标准进行组合评估时，结果是可靠的。
