Daly Caitlin H, Neupane Binod, Beyene Joseph, Thabane Lehana, Straus Sharon E, Hamid Jemila S
Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, Ontario, Canada.
Department of Mathematics and Statistics, McMaster University, Hamilton, Ontario, Canada.
BMJ Open. 2019 Sep 5;9(9):e024625. doi: 10.1136/bmjopen-2018-024625.
To provide a framework for quantifying the robustness of treatment ranks based on Surface Under the Cumulative RAnking curve (SUCRA) in network meta-analysis (NMA) and investigating potential factors associated with lack of robustness.
We propose the use of Cohen's kappa to quantify the agreement between SUCRA-based treatment ranks estimated through NMA of a complete data set and a subset of it. We illustrate our approach using five published NMA data sets, where robustness was assessed by removing studies one at a time.
Overall, SUCRA-based treatment ranks were robust to individual studies in the five data sets we considered. We observed more incidences of disagreement between ranks in the networks with larger numbers of treatments. Most treatments moved only one or two ranks up or down. The lowest quadratic weighted kappa estimate observed across all networks was in the network with the smallest number of treatments (4), where weighted kappa=40%. In the network with the largest number of treatments (12), the lowest observed quadratic weighted kappa=89%, reflecting a small shift in this network's treatment ranks overall. Preliminary observations suggest that a study's size, the number of studies making a treatment comparison, and the agreement of a study's estimated treatment effect(s) with those estimated by other studies making the same comparison(s) may explain the overall robustness of treatment ranks to studies.
Investigating robustness or sensitivity in an NMA may reveal outlying rank changes that are clinically or policy-relevant. Cohen's kappa is a useful measure that permits investigation into study characteristics that may explain varying sensitivity to individual studies. However, this study presents a framework as a proof of concept and further investigation is required to identify potential factors associated with the robustness of treatment ranks using more extensive empirical evaluations.
提供一个框架，用于在网络荟萃分析（NMA）中基于累积排序曲线下面积（SUCRA）量化治疗排序的稳健性，并调查与缺乏稳健性相关的潜在因素。
我们建议使用科恩kappa系数来量化通过完整数据集及其子集的NMA估计的基于SUCRA的治疗排序之间的一致性。我们使用五个已发表的NMA数据集来说明我们的方法，通过一次移除一项研究来评估稳健性。
总体而言，在我们考虑的五个数据集中，基于SUCRA的治疗排序对个别研究具有稳健性。我们观察到，治疗数量较多的网络中排序之间的不一致情况更多。大多数治疗仅上升或下降一两个排序。在所有网络中观察到的最低二次加权kappa估计值出现在治疗数量最少（4种）的网络中，其中加权kappa = 40%。在治疗数量最多（12种）的网络中，观察到的最低二次加权kappa = 89%，反映出该网络总体治疗排序的微小变化。初步观察表明，一项研究的规模、进行治疗比较的研究数量以及该研究估计的治疗效果与进行相同比较的其他研究估计的效果之间的一致性，可能解释了治疗排序对研究的总体稳健性。
在NMA中研究稳健性或敏感性可能会揭示临床上或政策上相关的异常排序变化。科恩kappa系数是一种有用的度量方法，可用于研究可能解释对个别研究敏感性差异的研究特征。然而，本研究提出了一个作为概念验证的框架，需要进一步研究以通过更广泛的实证评估来确定与治疗排序稳健性相关的潜在因素。
