Department of Educational and Psychological Studies, School of Education and Human Development, University of Miami, Miami, Florida, United States of America.
Sylvester Comprehensive Cancer Center, Leonard M. Miller School of Medicine, University of Miami, Miami, Florida, United States of America.
PLoS One. 2023 Jul 26;18(7):e0284469. doi: 10.1371/journal.pone.0284469. eCollection 2023.
Most existing quality scales have been developed with minimal attention to accepted standards of psychometric properties. Even for those that have been used widely in medical research, limited evidence exists supporting their psychometric properties. The focus of our current study is to address this gap by evaluating the psychometrics properties of two existing quality scales that are frequently used in cancer observational research: (1) Item Bank on Risk of Bias and Precision of Observational Studies developed by the Research Triangle Institute (RTI) International and (2) Newcastle-Ottawa Quality Assessment Scale (NOQAS). We used the Rasch measurement model to evaluate the psychometric properties of two quality scales based on the ratings of 49 studies that examine firefighters' cancer incidence and mortality. Our study found that RTI and NOQAS have an acceptable item reliability. Two raters were consistent in their assessment, demonstrating high interrater reliability. We also found that NOQAS has more items that show better fit than the RTI scale. The NOQAS produced lower study quality scores with a smaller variation, suggesting that NOQAS items are much easier to rate. Our findings accord with a previous study, which conclude that the RTI scale was harder to apply and thus produces more heterogenous quality scores than NOQAS. Although both RTI and NOQAS showed high item reliability, NOQAS items are better fit to the underlying construct, showing higher validity of internal structure and stronger psychometric properties. The current study adds to our understanding of the psychometric properties of NOQAS and RTI scales for future meta-analyses of observational studies, particularly in the firefighter cancer literature.
大多数现有的质量量表在开发过程中很少关注心理测量学属性的公认标准。即使是那些在医学研究中广泛使用的量表，也只有有限的证据支持其心理测量学特性。我们当前研究的重点是通过评估两种在癌症观察性研究中经常使用的现有质量量表的心理测量学特性来解决这一差距：（1）由三角研究所（RTI）国际开发的风险偏倚和观察研究精度项目库；（2）纽卡斯尔-渥太华质量评估量表（NOQAS）。我们使用 Rasch 测量模型，根据 49 项研究对消防员癌症发病率和死亡率的评估结果，评估了两种质量量表的心理测量学特性。我们的研究发现，RTI 和 NOQAS 的项目可靠性可以接受。两名评分者在评估方面具有一致性，表现出较高的评分者间可靠性。我们还发现，NOQAS 比 RTI 量表有更多的项目表现出更好的拟合度。NOQAS 产生的研究质量评分较低，且变化较小，这表明 NOQAS 项目更容易评分。我们的研究结果与之前的一项研究结果一致，该研究得出的结论是，RTI 量表更难应用，因此产生的质量评分比 NOQAS 更不一致。虽然 RTI 和 NOQAS 都表现出较高的项目可靠性，但 NOQAS 项目更符合潜在的结构，表现出更高的内部结构有效性和更强的心理测量学特性。本研究增加了我们对 NOQAS 和 RTI 量表心理测量学特性的理解，为未来观察性研究的荟萃分析，特别是在消防员癌症文献中提供了依据。
