Peeters Michael J, Cor M Kenneth, Petite Sarah E, Schroeder Michelle N
University of Toledo College of Pharmacy & Pharmaceutical Sciences, Toledo, OH.
University of Alberta Faculty of Pharmacy & Pharmaceutical Sciences, Edmonton, AB.
Innov Pharm. 2021 Feb 26;12(1). doi: 10.24926/iip.v12i1.2110. eCollection 2021.
Performance-based assessments, including objective structured clinical examinations (OSCEs), are essential learning assessments within pharmacy education. Because important educational decisions can follow from performance-based assessment results, pharmacy colleges/schools should demonstrate acceptable rigor in validation of their learning assessments. Though G-Theory has rarely been reported in pharmacy education, it would behoove pharmacy educators to, using G-Theory, produce evidence demonstrating reliability as a part of their OSCE validation process. This investigation demonstrates the use of G-Theory to describes reliability for an OSCE, as well as to show methods for enhancement of the OSCE's reliability.
To evaluate practice-readiness in the semester before final-year rotations, third-year PharmD students took an OSCE. This OSCE included 14 stations over three weeks. Each week had four or five stations; one or two stations were scored by faculty-raters while three stations required students' written responses. All stations were scored 1-4. For G-Theory analyses, we used G_Strings and then mGENOVA.
Ninety-seven students completed the OSCE; stations were scored independently. First, univariate G-Theory design of students crossed with stations nested in weeks (p × s:w) was used. The total-score g-coefficient (reliability) for this OSCE was 0.72. Variance components for test parameters were identified. Of note, students accounted for only some OSCE score variation. Second, a multivariate G-Theory design of students crossed with stations (p × s°) was used. This further analysis revealed which week(s) were weakest for the reliability of test-scores from this learning assessment. Moreover, decision-studies showed how reliability could change depending on the number of stations each week. For a g-coefficient >0.80, seven stations per week were needed. Additionally, targets for improvements were identified.
In test validation, evidence of reliability is vital for the inference of generalization; G-Theory provided this for our OSCE. Results indicated that the reliability of scores was mediocre and could be improved with more stations. Revision of problematic stations could help reliability as well. Within this need for more stations, one practical insight was to administer those stations over multiple weeks/occasions (instead of all stations in one occasion).
基于表现的评估，包括客观结构化临床考试（OSCE），是药学教育中至关重要的学习评估方式。由于基于表现的评估结果可能会引发重要的教育决策，药学院校应在其学习评估的验证过程中展现出可接受的严谨性。尽管G理论在药学教育中的报道很少，但药学教育工作者使用G理论来提供证据证明可靠性，作为其OSCE验证过程的一部分是很有必要的。本研究展示了如何使用G理论来描述OSCE的可靠性，以及提高OSCE可靠性的方法。
为了评估在最后一年轮转前一个学期的实践准备情况，三年级药学博士学生参加了一次OSCE。这次OSCE在三周内设有14个考站。每周有四或五个考站；一两个考站由教师评分员评分，而三个考站要求学生进行书面回答。所有考站的评分都是1 - 4分。对于G理论分析，我们使用了G_Strings，然后是mGENOVA。
97名学生完成了OSCE；考站由不同的人独立评分。首先，采用了学生与嵌套在周内的考站的单变量G理论设计（p×s:w）。这次OSCE的总分g系数（可靠性）为0.72。确定了测试参数的方差成分。值得注意的是，学生只占OSCE分数变化的一部分。其次，采用了学生与考站的多变量G理论设计（p×s°）。进一步的分析揭示了从这次学习评估中测试分数可靠性最差的是哪一周。此外，决策研究表明可靠性如何根据每周考站的数量而变化。为了使g系数>0.80，每周需要七个考站。此外，还确定了改进的目标。
在测试验证中，可靠性证据对于推广性推断至关重要；G理论为我们的OSCE提供了这一证据。结果表明分数的可靠性一般，可以通过增加考站数量来提高。对有问题的考站进行修订也有助于提高可靠性。在需要更多考站的情况下，一个实际的见解是在多个周/场次进行这些考站的测试（而不是在一次考试中设置所有考站）。
