Faculty of Health, Zuyd University of Applied Sciences, Nieuw Eyckholt 300, 6419, DJ, Heerlen, The Netherlands.
Department of Educational Development and Research, Maastricht University, Universiteitssingel 60, 6229, ER, Maastricht, The Netherlands.
BMC Med Educ. 2017 Sep 5;17(1):151. doi: 10.1186/s12909-017-0989-x.
Items must be relevant to ensure item quality and test validity. Since "item relevance" has not been operationalized yet, we developed a rubric to define it. This study explores the influence of this rubric on the assessment of item relevance and on inter-rater agreement.
Members of the item review committee (RC) and students, teachers, and alumni (STA) reassessed the relevance of 50 previously used progress test (PT) items and decided about their inclusion using a 5-criteria rubric. Data were analyzed at item level using paired samples t-tests, Intraclass Correlation Coefficients (ICC), and linear regression analysis, and at rater level in a generalizability analysis per group.
The proportion of items that the RC judged relevant enough to be included decreased substantially from 1.00 to 0.72 (p < 0.001). Agreement between the RC and STA was high, with an ICC of >0.7 across items. The relation between inclusion and relevance was strong (correlation = 0.89, p < 0.001), and did not differ between RC and STA. To achieve an acceptable inter-rater reliability for relevance and inclusion, 6 members must serve on the RC.
Use of the rubric results in a stricter evaluation of items' appropriateness for inclusion in the PT and facilitates agreement between the RC and other stakeholders. Hence, it may help increase the acceptability and validity of the PT.
项目必须相关，以确保项目质量和测试有效性。由于“项目相关性”尚未付诸实践，我们开发了一个评分标准来定义它。本研究探讨了该评分标准对项目相关性评估和评分者间一致性的影响。
项目审查委员会（RC）成员和学生、教师和校友（STA）重新评估了 50 项先前使用的进展测试（PT）项目的相关性，并使用 5 项标准评分标准决定是否纳入。使用配对样本 t 检验、组内相关系数（ICC）和线性回归分析在项目水平上对数据进行分析，并在每个组的可概括性分析中在评分者水平上进行分析。
RC 认为足够相关以纳入的项目比例从 1.00 大幅下降到 0.72（p<0.001）。RC 和 STA 之间的一致性很高，ICC 大于 0.7。纳入和相关性之间的关系很强（相关系数为 0.89，p<0.001），RC 和 STA 之间没有差异。为了实现相关性和纳入的可接受评分者间可靠性，RC 必须有 6 名成员。
使用评分标准对项目纳入 PT 的适当性进行更严格的评估，并促进 RC 和其他利益相关者之间的一致性。因此，它可能有助于提高 PT 的可接受性和有效性。
