Turbow David J, Evener Julie
J Med Libr Assoc. 2016 Jul;104(3):209-14. doi: 10.3163/1536-5050.104.3.005.
The study evaluated whether a modified version of the information literacy Valid Assessment of Learning in Undergraduate Education (VALUE) rubric would be useful for assessing the information literacy skills of graduate health sciences students.
Through facilitated calibration workshops, an interdepartmental six-person team of librarians and faculty engaged in guided discussion about the meaning of the rubric criteria. They applied the rubric to score student work for a peer-review essay assignment in the "Information Literacy for Evidence-Based Practice" course. To determine inter-rater reliability, the raters participated in a follow-up exercise in which they independently applied the rubric to ten samples of work from a research project in the doctor of physical therapy program: the patient case report assignment.
For the peer-review essay, a high level of consistency in scoring was achieved for the second workshop, with statistically significant intra-class correlation coefficients above 0.8 for 3 criteria: "Determine the extent of evidence needed," "Use evidence effectively to accomplish a specific purpose," and "Access the needed evidence." Participants concurred that the essay prompt and rubric criteria adequately discriminated the quality of student work for the peer-review essay assignment. When raters independently scored the patient case report assignment, inter-rater agreement was low and statistically insignificant for all rubric criteria (kappa=-0.16, p>0.05-kappa=0.12, p>0.05).
While the peer-review essay assignment lent itself well to rubric calibration, scorers had a difficult time with the patient case report. Lack of familiarity among some raters with the specifics of the patient case report assignment and subject matter might have accounted for low inter-rater reliability. When norming, it is important to hold conversations about search strategies and expectations of performance. Overall, the authors found the rubric to be appropriate for assessing information literacy skills of graduate health sciences students.
本研究评估本科教育学习有效评估（VALUE）信息素养评分标准的修订版是否有助于评估健康科学专业研究生的信息素养技能。
通过开展校准研讨会，一个由图书馆员和教师组成的跨部门六人团队就评分标准的含义进行了引导性讨论。他们将该评分标准应用于为“循证实践的信息素养”课程中的一篇同行评审论文作业的学生作品打分。为了确定评分者间的可靠性，评分者参与了一项后续活动，他们独立地将评分标准应用于物理治疗博士项目中一个研究项目的十个作品样本：患者病例报告作业。
对于同行评审论文，第二次研讨会在评分方面实现了高度一致性，3个标准的组内相关系数在统计上显著高于0.8，这3个标准分别是：“确定所需证据的范围”、“有效利用证据实现特定目的”以及“获取所需证据”。参与者一致认为论文提示和评分标准能够充分区分同行评审论文作业中学生作品的质量。当评分者独立为患者病例报告作业打分时，所有评分标准的评分者间一致性较低且在统计上不显著（kappa=-0.16，p>0.05——kappa=0.12，p>0.05）。
虽然同行评审论文作业很适合进行评分标准校准，但评分者在患者病例报告方面遇到了困难。一些评分者对患者病例报告作业和主题内容的具体细节缺乏熟悉度，这可能是评分者间可靠性较低的原因。在进行规范时，就搜索策略和表现期望进行讨论很重要。总体而言，作者发现该评分标准适用于评估健康科学专业研究生的信息素养技能。
