Department of Radiology, University of Pittsburgh, Pittsburgh, PA 15213, USA.
Acad Radiol. 2010 May;17(5):628-38. doi: 10.1016/j.acra.2010.01.007.
Sample-size estimation is an important consideration when planning a receiver operating characteristic (ROC) study. The aim of this work was to assess the prediction accuracy of a sample-size estimation method using the Monte Carlo simulation method.
Two ROC ratings simulators characterized by low reader and high case variabilities (LH) and high reader and low case variabilities (HL) were used to generate pilot data sets in two modalities. Dorfman-Berbaum-Metz multiple-reader multiple-case (DBM-MRMC) analysis of the ratings yielded estimates of the modality-reader, modality-case, and error variances. These were input to the Hillis-Berbaum (HB) sample-size estimation method, which predicted the number of cases needed to achieve 80% power for 10 readers and an effect size of 0.06 in the pivotal study. Predictions that generalized to readers and cases (random-all), to cases only (random-cases), and to readers only (random-readers) were generated. A prediction-accuracy index defined as the probability that any single prediction yields true power in the 75%-90% range was used to assess the HB method.
For random-case generalization, the HB-method prediction-accuracy was reasonable, approximately 50% for five readers and 100 cases in the pilot study. Prediction-accuracy was generally higher under LH conditions than under HL conditions. Under ideal conditions (many readers in the pilot study) the DBM-MRMC-based HB method overestimated the number of cases. The overestimates could be explained by the larger modality-reader variance estimates when reader variability was large (HL). The largest benefit of increasing the number of readers in the pilot study was realized for LH, where 15 readers were enough to yield prediction accuracy >50% under all generalization conditions, but the benefit was lesser for HL where prediction accuracy was approximately 36% for 15 readers under random-all and random-reader conditions.
The HB method tends to overestimate the number of cases. Random-case generalization had reasonable prediction accuracy. Provided about 15 readers were used in the pilot study the method performed reasonably under all conditions for LH. When reader variability was large, the prediction-accuracy for random-all and random-reader generalizations was compromised. Study designers may wish to compare the HB predictions to those of other methods and to sample-sizes used in previous similar studies.
在规划受试者工作特征（ROC）研究时，样本量估计是一个重要的考虑因素。本研究旨在使用蒙特卡罗模拟方法评估一种样本量估计方法的预测准确性。
使用两种 ROC 评分模拟器，其特征为低读者和高病例变异性（LH）以及高读者和低病例变异性（HL），在两种模态下生成试点数据集。Dorfman-Berbaum-Metz 多位读者多位病例（DBM-MRMC）分析对评分进行了分析，得出了模态-读者、模态-病例和误差方差的估计值。这些值被输入到 Hillis-Berbaum（HB）样本量估计方法中，该方法预测了在关键研究中 10 位读者和 0.06 效应大小的情况下，实现 80%功效所需的病例数。生成了对读者和病例进行泛化（随机-所有）、仅对病例进行泛化（随机-病例）以及仅对读者进行泛化（随机-读者）的预测。使用定义为任何单个预测在 75%-90%范围内产生真实功效的概率的预测准确性指数来评估 HB 方法。
对于随机病例泛化，HB 方法的预测准确性是合理的，在试点研究中，对于 5 位读者和 100 个病例，预测准确性约为 50%。在 LH 条件下，预测准确性通常高于 HL 条件。在理想条件下（试点研究中有许多读者），基于 DBM-MRMC 的 HB 方法高估了病例数。当读者变异性较大（HL）时，较大的模态-读者方差估计导致了高估。在试点研究中增加读者数量的最大益处是在 LH 中实现的，其中 15 位读者足以在所有泛化条件下产生>50%的预测准确性，但对于 HL 来说，这种益处较小，其中在随机-所有和随机-读者条件下，15 位读者的预测准确性约为 36%。
HB 方法倾向于高估病例数。随机病例泛化具有合理的预测准确性。只要在试点研究中使用大约 15 位读者，该方法在 LH 下的所有条件下表现合理。当读者变异性较大时，随机-所有和随机-读者泛化的预测准确性会受到影响。研究设计者可能希望将 HB 预测与其他方法的预测进行比较，并与以前类似研究中使用的样本量进行比较。
