Allen Coburn, Deanehan J Kate, Dotan Yaniv, Eisenberg Matthew A, Fine Andrew M, Isenberg Jonathan, Kane Ann, Kirshner Dani, Lyons Todd W, Maor Yasmin, Neuberger Ami, Ostermayer Daniel G, Paz Sharona, Scheuerman Oded, Shiber Shachaf, Statler Victoria A, Stein Michal, Yakubov Renata, Yanai Shirly, Navon Roy, Kellerman Lior, Gottlieb Tanya M, Eden Eran
Department of Pediatrics, University of Texas at Austin Dell Medical School, Austin, Texas, USA.
Department of Pediatric Emergency Medicine, John's Hopkins Children's Center, Baltimore, Maryland, USA.
Clin Infect Dis. 2025 Apr 30;80(4):735-743. doi: 10.1093/cid/ciae656.
Diagnostic test evaluation requires a reference standard. We describe an approach for creating a reference standard for acute infection using unrestricted adjudication and apply it to compare biomarker tools.
Adults and children with suspected acute infection enrolled in three prospective studies at emergency departments and urgent cares were included. Adjudicators, blinded to C-reactive protein, procalcitonin, and MeMed BV (MMBV), labeled each case (bacterial/viral/non-infectious/indeterminate). Initial adjudication involved 3 adjudicators. Reference standard cohorts were defined: Microbiologically confirmed (3/3 adjudicators concur with high confidence and a concordant microbiological finding), unanimous (3/3 adjudicators concur with high confidence), suspected (3/3 adjudicators concur with high/moderate confidence or 2/3 adjudicators concur with high confidence), and all-inclusive (remaining unlabeled cases were reviewed by up to 7 additional adjudicators until reaching a leading label).
Among 1016 patients, 156 difficult-to-diagnose cases required over 3 adjudicators. The area under the receiver operating characteristic curve in the microbiologically confirmed (n = 427), unanimous (n = 565), suspected (n = 860), and all-inclusive (n = 1016) cohorts for MMBV were 0.98 (95% confidence interval .94-1.00), 0.98 (.95-1.00), 0.95 (.92-.98) and 0.90 (.87-.93), respectively, and for procalcitonin were 0.69 (.57-.81), 0.77 (.68-.86), 0.74 (.68-.80) and 0.70 (.65-.75), respectively. A delta in performance between MMBV and procalcitonin was maintained across the different cohorts.
Creating a reference standard that includes difficult-to-diagnose cases demands an approach to addressing diagnostic uncertainty in acute infections. Tool performance depends on the reference standard applied and decreases as the difficulty to diagnose increases, highlighting the importance of using the same reference standard when comparing tools.
诊断试验评估需要一个参考标准。我们描述了一种使用无限制判定法为急性感染创建参考标准的方法，并将其应用于比较生物标志物工具。
纳入在急诊科和紧急护理中心参加三项前瞻性研究的疑似急性感染的成人和儿童。判定人员在不知道C反应蛋白、降钙素原和MeMed BV（MMBV）结果的情况下，对每个病例进行标记（细菌感染/病毒感染/非感染性/不确定）。初始判定由3名判定人员进行。定义了参考标准队列：微生物学确诊（3名判定人员高度一致且有一致的微生物学检查结果）、一致判定（3名判定人员高度一致）、疑似（3名判定人员高度/中度一致或2名判定人员高度一致）和全面涵盖（其余未标记病例由多达7名额外判定人员进行审查，直至得出主要标记）。
在1016例患者中，156例难以诊断的病例需要超过3名判定人员进行判定。MMBV在微生物学确诊队列（n = 427）、一致判定队列（n = 565）、疑似队列（n = 860）和全面涵盖队列（n = 1016）中的受试者操作特征曲线下面积分别为0.98（95%置信区间0.94 - 1.00）、0.98（0.95 - 1.00）、0.95（0.92 - 0.98）和0.90（0.87 - 0.93），降钙素原的相应值分别为0.69（0.57 - 0.81）、0.77（0.68 - 0.86）、0.74（0.68 - 0.80）和0.70（0.65 - 0.75）。MMBV和降钙素原之间的性能差异在不同队列中均保持。
创建一个包括难以诊断病例的参考标准需要一种解决急性感染诊断不确定性的方法。工具性能取决于所应用的参考标准，并且随着诊断难度的增加而降低，这突出了在比较工具时使用相同参考标准的重要性。
