Kellerhuis B E, Jenniskens K, Schuit E, Hooft L, Moons K G M, Reitsma J B
Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht, The Netherlands.
BMC Med Res Methodol. 2025 Apr 23;25(1):106. doi: 10.1186/s12874-025-02557-7.
Expert panels are often used as a reference standard when no gold standard is available in diagnostic test accuracy research. It is often unclear what study and expert panel characteristics produce the best estimates of diagnostic test accuracy. We simulated a large range of scenarios to assess the impact of study and expert panel characteristics on index test diagnostic accuracy estimates.
Simulations were performed in which an expert panel was the reference standard to estimate the sensitivity and specificity of an index diagnostic test. Diagnostic accuracy was determined by combining probability estimates of target condition presence, provided by experts using four component reference tests, through a predefined threshold. Study and panel characteristics were varied in several scenarios: target condition prevalence, accuracy of component reference tests, expert panel size, study population size, and random or systematic differences between expert's probability estimates. The total bias in each scenario was quantified using mean squared error.
When estimating an index test with 80% sensitivity and 70% specificity, bias in estimates was hardly affected by the study population size or the number of experts. Prevalence had a large effect on bias, scenarios with a prevalence of 0.5 estimated sensitivity between 63.3% and 76.7% and specificity between 56.1% and 68.7%, whereas scenarios with a prevalence of 0.2 estimated sensitivity between 48.5% and 73.3% and specificity between 65.5% and 68.7%. Improved reference tests also reduced bias. Scenarios with four component tests of 80% sensitivity and specificity estimated index test sensitivity between 60.1% and 77.4% and specificity between 62.9% and 69.1%, whereas scenarios with four component tests of 70% sensitivity and specificity estimated index test sensitivity between 48.5% and 73.4% and specificity between 56.1% and 67.0%.
Bias in accuracy estimates when using an expert panel will increase if the component reference tests are less accurate. Prevalence, the true value of the index test accuracy, and random or systematic differences between experts can also impact the amount of bias, but the amount and even direction will vary between scenarios.
在诊断试验准确性研究中，当没有金标准可用时，专家小组常被用作参考标准。通常不清楚哪些研究和专家小组特征能产生对诊断试验准确性的最佳估计。我们模拟了大量场景，以评估研究和专家小组特征对指标试验诊断准确性估计的影响。
进行模拟，其中专家小组作为参考标准来估计指标诊断试验的敏感性和特异性。诊断准确性通过使用四个组成参考试验由专家提供的目标疾病存在概率估计值，通过预定义阈值进行组合来确定。在几种场景中改变研究和小组特征：目标疾病患病率、组成参考试验的准确性、专家小组规模、研究人群规模以及专家概率估计值之间的随机或系统差异。使用均方误差对每个场景中的总偏差进行量化。
在估计敏感性为80%、特异性为70%的指标试验时，估计偏差几乎不受研究人群规模或专家数量的影响。患病率对偏差有很大影响，患病率为0.5的场景估计敏感性在63.3%至76.7%之间，特异性在56.1%至68.7%之间，而患病率为0.2的场景估计敏感性在48.5%至73.3%之间，特异性在65.5%至68.7%之间。改进的参考试验也减少了偏差。四个组成试验敏感性和特异性均为80%的场景估计指标试验敏感性在60.1%至77.4%之间，特异性在62.9%至69.1%之间，而四个组成试验敏感性和特异性均为70%的场景估计指标试验敏感性在48.5%至73.4%之间，特异性在56.1%至67.0%之间。
如果组成参考试验准确性较低，使用专家小组时准确性估计中的偏差将会增加。患病率、指标试验准确性的真实值以及专家之间的随机或系统差异也会影响偏差量，但偏差量甚至方向在不同场景之间会有所不同。
