Fulton Lawrence, Lieneck Cristian, Ramamonjiarivelo Zo, Kruse Clemens Scott, Brooks Matthew S
Department of Health Administration, 601 University Drive, Texas State University, San Marcos, TX, 78666, USA.
BMC Med Educ. 2021 Jan 6;21(1):21. doi: 10.1186/s12909-020-02448-z.
Assessing competencies or program learning outcomes in educational programs is often a leadership challenge. This case study reports medical education program's efforts to document undergraduate competency attainment using a pre-post, third-party, objective testing service that allows for inter-university comparison, a testing service that is being adopted by some certification and accrediting bodies.
Students completed a pre-test after program acceptance and a post-test at the end of the last didactic semester (1.5 years later) just prior to their required internships. Scores and subscores were evaluated using t-tests (Holm-adjusted p-values). MANOVA models of sub-competency difference scores were also evaluated.
Results indicate competency improvement for each of the 12 areas based on the n = 55 student sample, (p < .001 for all scores). These improvements were independent of ethnicity, age, gender, and grades. The average student improved by 12.85 points (95% CI of 10.52 to 15.18) with the largest improvements in strategic planning and leadership competency areas (21.30 and 18.33 percentage points, respectively).
The third-party pre-post has some face validity given that student performance improved after completing a related curriculum as would be expected. Congruent with earlier studies, we find that repeated testing helps document competency attainment and that a single method for assessment is insufficient. We further document limitations of this 3d-party exam.
在教育项目中评估能力或项目学习成果往往是一项领导力挑战。本案例研究报告了一个医学教育项目为记录本科学生能力达成情况所做的努力，该项目使用了一种前后测、第三方客观测试服务，这种服务允许进行校际比较，一些认证和评审机构也正在采用这种测试服务。
学生在被项目录取后完成了一次预测试，并在最后一个理论学期末（1.5年后）、在他们必修实习之前完成了一次后测试。使用t检验（经霍尔姆调整的p值）对分数和子分数进行评估。还对子能力差异分数的多变量方差分析模型进行了评估。
基于n = 55名学生的样本，结果表明12个领域中的每个领域能力都有所提高（所有分数的p <.001）。这些提高与种族、年龄、性别和成绩无关。平均每个学生提高了12.85分（95%置信区间为10.52至15.18），其中战略规划和领导能力领域的提高最大（分别为21.30和18.33个百分点）。
鉴于学生在完成相关课程后成绩如预期那样提高，第三方前后测具有一定的表面效度。与早期研究一致，我们发现重复测试有助于记录能力达成情况，且单一的评估方法是不够的。我们还记录了这种第三方考试的局限性。
