Harvard School of Public Health, Boston, Massachusetts, USA.
BMC Med Inform Decis Mak. 2010 May 7;10:25. doi: 10.1186/1472-6947-10-25.
Syndromic surveillance systems can potentially be used to detect a bioterrorist attack earlier than traditional surveillance, by virtue of their near real-time analysis of relevant data. Receiver operator characteristic (ROC) curve analysis using the area under the curve (AUC) as a comparison metric has been recommended as a practical evaluation tool for syndromic surveillance systems, yet traditional ROC curves do not account for timeliness of detection or subsequent time-dependent health outcomes.
Using a decision-analytic approach, we predicted outcomes, measured in lives, quality adjusted life years (QALYs), and costs, for a series of simulated bioterrorist attacks. We then evaluated seven detection algorithms applied to syndromic surveillance data using outcomes-weighted ROC curves compared to simple ROC curves and timeliness-weighted ROC curves. We performed sensitivity analyses by varying the model inputs between best and worst case scenarios and by applying different methods of AUC calculation.
The decision analytic model results indicate that if a surveillance system was successful in detecting an attack, and measures were immediately taken to deliver treatment to the population, the lives, QALYs and dollars lost could be reduced considerably. The ROC curve analysis shows that the incorporation of outcomes into the evaluation metric has an important effect on the apparent performance of the surveillance systems. The relative order of performance is also heavily dependent on the choice of AUC calculation method.
This study demonstrates the importance of accounting for mortality, morbidity and costs in the evaluation of syndromic surveillance systems. Incorporating these outcomes into the ROC curve analysis allows for more accurate identification of the optimal method for signaling a possible bioterrorist attack. In addition, the parameters used to construct an ROC curve should be given careful consideration.
综合征监测系统通过实时分析相关数据，有可能比传统监测更早地发现生物恐怖袭击。使用接收者操作特征（ROC）曲线分析，以曲线下面积（AUC）作为比较指标，已被推荐作为综合征监测系统的实用评估工具，但传统的 ROC 曲线没有考虑检测的及时性或随后的时间相关健康结果。
我们使用决策分析方法，预测了一系列模拟生物恐怖袭击的结果，以生命、质量调整生命年（QALY）和成本来衡量。然后，我们使用基于结果的 ROC 曲线和基于时间的 ROC 曲线评估了七种应用于综合征监测数据的检测算法，与简单的 ROC 曲线进行了比较。我们通过在最佳和最差情况下改变模型输入，并应用不同的 AUC 计算方法进行了敏感性分析。
决策分析模型的结果表明，如果监测系统成功检测到袭击，并且立即采取措施向人群提供治疗，那么失去的生命、QALY 和美元就可以大大减少。ROC 曲线分析表明，将结果纳入评估指标对监测系统的性能有重要影响。性能的相对顺序也严重依赖于 AUC 计算方法的选择。
本研究表明，在评估综合征监测系统时，考虑死亡率、发病率和成本非常重要。将这些结果纳入 ROC 曲线分析，可以更准确地确定提示可能发生生物恐怖袭击的最佳方法。此外，构建 ROC 曲线所使用的参数应仔细考虑。
