Helmholtz Zentrum, München, Ingolstädter Landstr. 1, Neuherberg, 85764, Germany.
Department of Medical Information Processing, Biometry and Epidemiology, University of Munich, Marchioninistr. 15, Munich, 81377, Germany.
Radiat Oncol. 2020 May 14;15(1):109. doi: 10.1186/s13014-020-01543-1.
Prognostic models based on high-dimensional omics data generated from clinical patient samples, such as tumor tissues or biopsies, are increasingly used for prognosis of radio-therapeutic success. The model development process requires two independent discovery and validation data sets. Each of them may contain samples collected in a single center or a collection of samples from multiple centers. Multi-center data tend to be more heterogeneous than single-center data but are less affected by potential site-specific biases. Optimal use of limited data resources for discovery and validation with respect to the expected success of a study requires dispassionate, objective decision-making. In this work, we addressed the impact of the choice of single-center and multi-center data as discovery and validation data sets, and assessed how this impact depends on the three data characteristics signal strength, number of informative features and sample size.
We set up a simulation study to quantify the predictive performance of a model trained and validated on different combinations of in silico single-center and multi-center data. The standard bioinformatical analysis workflow of batch correction, feature selection and parameter estimation was emulated. For the determination of model quality, four measures were used: false discovery rate, prediction error, chance of successful validation (significant correlation of predicted and true validation data outcome) and model calibration.
In agreement with literature about generalizability of signatures, prognostic models fitted to multi-center data consistently outperformed their single-center counterparts when the prediction error was the quality criterion of interest. However, for low signal strengths and small sample sizes, single-center discovery sets showed superior performance with respect to false discovery rate and chance of successful validation.
With regard to decision making, this simulation study underlines the importance of study aims being defined precisely a priori. Minimization of the prediction error requires multi-center discovery data, whereas single-center data are preferable with respect to false discovery rate and chance of successful validation when the expected signal or sample size is low. In contrast, the choice of validation data solely affects the quality of the estimator of the prediction error, which was more precise on multi-center validation data.
基于从临床患者样本（如肿瘤组织或活检）生成的高维组学数据的预后模型，越来越多地用于预测放射治疗的成功。模型开发过程需要两个独立的发现和验证数据集。每个数据集可能包含在单个中心收集的样本或来自多个中心的样本集合。多中心数据往往比单中心数据更具异质性，但受潜在的特定于地点的偏差影响较小。为了实现研究的预期成功，最佳地利用发现和验证的有限数据资源需要冷静、客观的决策。在这项工作中，我们研究了选择单中心和多中心数据作为发现和验证数据集对模型预测性能的影响，并评估了这种影响如何取决于三个数据特征：信号强度、信息量特征的数量和样本量。
我们设计了一个模拟研究，以量化在虚拟单中心和多中心数据的不同组合上训练和验证的模型的预测性能。模拟了批量校正、特征选择和参数估计的标准生物信息学分析工作流程。为了确定模型质量，使用了四个指标：假发现率、预测误差、成功验证的机会（预测和真实验证数据结果之间存在显著相关性）和模型校准。
与关于签名通用性的文献一致，当感兴趣的质量标准是预测误差时，拟合多中心数据的预后模型始终优于其单中心对应模型。然而，对于低信号强度和小样本量，单中心发现集在假发现率和成功验证的机会方面表现出更好的性能。
就决策而言，这项模拟研究强调了在事先明确定义研究目标的重要性。最小化预测误差需要多中心发现数据，而在预期信号或样本量较低时，单中心数据在假发现率和成功验证的机会方面更具优势。相比之下，验证数据的选择仅影响预测误差的估计质量，多中心验证数据的估计质量更精确。
