Del Paggio Joseph C, Cheng Sierra, Booth Christopher M, Cheung Matthew C, Chan Kelvin K W
Department of Medicine, Division of Medical Oncology, University of Toronto, Toronto, Ontario Canada.
Departments of Oncology and Public Health Sciences, Queen's University, Kingston, Ontario, Canada.
JNCI Cancer Spectr. 2018 Dec 13;2(3):pky050. doi: 10.1093/jncics/pky050. eCollection 2018 Jul.
Research groups are increasingly utilizing value frameworks, but little is known of their reliability. To assess framework concordance and interrater reliability between two major value frameworks currently in use, we identified all previously published datasets containing both scores from the American Society of Clinical Oncology Value Framework (ASCO-VF) and grades from the European Society for Medical Oncology-Magnitude of Clinical Benefit Scale (ESMO-MCBS). The intraclass correlation coefficient (ICC) was used to assess interrater reliability. Four eligible studies contained drugs evaluated by both value frameworks, resulting in a dataset of 39 grades/scores for discrete drug indications. ICC was 0.82 (95% confidence interval = 0.70 to 0.90) for ASCO-VF and 0.88 (95% confidence interval = 0.80 to 0.93) for ESMO-MCBS. Absolute concordance was found to be 5% for ASCO-VF and 44% for ESMO-MCBS, increasing to 74% and 80% when deviations within 20 points and 1 grade were considered, respectively. Interrater reliability of ASCO-VF and ESMO-MCBS is, therefore, near perfect, while absolute concordance is poor. This has implications when considering framework outputs in drug funding or treatment decision making.
研究小组越来越多地使用价值框架，但对其可靠性却知之甚少。为了评估目前正在使用的两个主要价值框架之间的框架一致性和评分者间信度，我们识别了所有先前发表的数据集，这些数据集既包含美国临床肿瘤学会价值框架（ASCO-VF）的评分，也包含欧洲医学肿瘤学会-临床获益程度量表（ESMO-MCBS）的分级。组内相关系数（ICC）用于评估评分者间信度。四项符合条件的研究包含了两个价值框架都评估过的药物，从而形成了一个包含39个离散药物适应症分级/评分的数据集。ASCO-VF的ICC为0.82（95%置信区间=0.70至0.90），ESMO-MCBS的ICC为0.88（95%置信区间=0.80至0.93）。发现ASCO-VF的绝对一致性为5%，ESMO-MCBS为44%，当分别考虑20分以内的偏差和1级以内的偏差时，一致性分别增至74%和80%。因此，ASCO-VF和ESMO-MCBS的评分者间信度接近完美，但绝对一致性较差。这在考虑药物资助或治疗决策中的框架输出时具有重要意义。
