Turner-Stokes Lynne, Fadyl Joanna, Rose Hilary, Williams Heather, Schlüter Philip, McPherson Kathryn
Department of Palliative Care, Policy and Rehabilitation, School of Medicine, King's College London, London, UK,
J Occup Rehabil. 2014 Sep;24(3):511-24. doi: 10.1007/s10926-013-9486-1.
The Work-ability Support Scale (WSS) is a new tool designed to assess vocational ability and support needs following onset of acquired disability, to assist decision-making in vocational rehabilitation. In this article, we report an iterative process of development through evaluation of inter- and intra-rater reliability and scoring accuracy, using vignettes. The impact of different methodological approaches to analysis of reliability is highlighted.
Following preliminary evaluation using case-histories, six occupational therapists scored vignettes, first individually and then together in two teams. Scoring was repeated blind after 1 month. Scoring accuracy was tested against agreed 'reference standard' vignette scores using intraclass correlation coefficients (ICCs) for total scores and linear-weighted kappas (kw) for individual items. Item-by-item inter- and intra-rater reliability was evaluated for both individual and team scores, using two different statistical methods.
ICCs for scoring accuracy ranged from 0.95 (95 % CI 0.78-0.98) to 0.96 (0.89-0.99) for Part A, and from 0.78 (95 % CI 0.67-0.85) to 0.84 (0.69-0.92) for Part B. Item by item analysis of scoring accuracy, inter- and intra-rater reliability all showed 'substantial' to 'almost perfect' agreement (kw ≥ 0.60) for all Part-A and 8/12 Part-B items, although multi-rater kappa (Fleiss) produced more conservative results (mK = 0.34-0.79). Team rating produced marginal improvements for Part-A but not Part-B. Four problematic contextual items were identified, leading to adjustment of the scoring manual.
This vignette-based study demonstrates generally acceptable levels of scoring accuracy and reliability for the WSS. Further testing in real-life situations is now warranted.
工作能力支持量表（WSS）是一种新工具，旨在评估后天性残疾发病后的职业能力和支持需求，以协助职业康复中的决策制定。在本文中，我们报告了一个通过使用案例 vignettes 评估评分者间和评分者内信度以及评分准确性的迭代开发过程。强调了不同可靠性分析方法的影响。
在使用病例史进行初步评估之后，六名职业治疗师对 vignettes 进行评分，首先单独评分，然后分成两个小组一起评分。1 个月后进行盲态重复评分。使用总分的组内相关系数（ICC）和单个项目的线性加权卡帕（kw），对照商定的“参考标准”vignette 评分来测试评分准确性。使用两种不同的统计方法评估单个和小组评分的逐项评分者间和评分者内信度。
A 部分评分准确性的 ICC 范围为 0.95（95%CI 0.78 - 0.98）至 0.96（0.89 - 0.99），B 部分为 0.78（95%CI 0.67 - 0.85）至 0.84（0.69 - 0.92）。对所有 A 部分和 12 个 B 部分项目中的 8 个项目进行的评分准确性、评分者间和评分者内信度的逐项分析均显示出“实质性”至“几乎完美”的一致性（kw≥0.60），尽管多评分者卡帕（Fleiss）产生的结果更为保守（mK = 0.34 - 0.79）。小组评分对 A 部分有轻微改善，但对 B 部分没有。识别出四个有问题的背景项目，导致对评分手册进行调整。
这项基于 vignettes 的研究表明，WSS 的评分准确性和信度总体上处于可接受水平。现在有必要在实际情况中进行进一步测试。
