School of Epidemiology and Public Health, University of Ottawa, Ottawa, ON, Canada.
Replica Analytics, Ottawa, ON, Canada.
Sci Rep. 2024 Mar 24;14(1):6978. doi: 10.1038/s41598-024-57207-7.
Synthetic data generation is being increasingly used as a privacy preserving approach for sharing health data. In addition to protecting privacy, it is important to ensure that generated data has high utility. A common way to assess utility is the ability of synthetic data to replicate results from the real data. Replicability has been defined using two criteria: (a) replicate the results of the analyses on real data, and (b) ensure valid population inferences from the synthetic data. A simulation study using three heterogeneous real-world datasets evaluated the replicability of logistic regression workloads. Eight replicability metrics were evaluated: decision agreement, estimate agreement, standardized difference, confidence interval overlap, bias, confidence interval coverage, statistical power, and precision (empirical SE). The analysis of synthetic data used a multiple imputation approach whereby up to 20 datasets were generated and the fitted logistic regression models were combined using combining rules for fully synthetic datasets. The effects of synthetic data amplification were evaluated, and two types of generative models were used: sequential synthesis using boosted decision trees and a generative adversarial network (GAN). Privacy risk was evaluated using a membership disclosure metric. For sequential synthesis, adjusted model parameters after combining at least ten synthetic datasets gave high decision and estimate agreement, low standardized difference, as well as high confidence interval overlap, low bias, the confidence interval had nominal coverage, and power close to the nominal level. Amplification had only a marginal benefit. Confidence interval coverage from a single synthetic dataset without applying combining rules were erroneous, and statistical power, as expected, was artificially inflated when amplification was used. Sequential synthesis performed considerably better than the GAN across multiple datasets. Membership disclosure risk was low for all datasets and models. For replicable results, the statistical analysis of fully synthetic data should be based on at least ten generated datasets of the same size as the original whose analyses results are combined. Analysis results from synthetic data without applying combining rules can be misleading. Replicability results are dependent on the type of generative model used, with our study suggesting that sequential synthesis has good replicability characteristics for common health research workloads.
合成数据生成正被越来越多地用作共享健康数据的隐私保护方法。除了保护隐私之外，确保生成的数据具有高实用性也很重要。评估实用性的一种常见方法是合成数据复制真实数据结果的能力。可重复性已使用两个标准定义：(a) 复制真实数据上的分析结果，以及 (b) 确保从合成数据中进行有效的总体推断。使用三个异构真实世界数据集进行的模拟研究评估了逻辑回归工作负载的可重复性。评估了八种可重复性指标：决策一致性、估计一致性、标准化差异、置信区间重叠、偏差、置信区间覆盖、统计功效和精度（经验 SE）。合成数据的分析使用了多次插补方法，最多可以生成 20 个数据集，并使用完全合成数据集的组合规则来组合拟合的逻辑回归模型。评估了合成数据放大的效果，并使用了两种生成模型：使用提升决策树的顺序合成和生成对抗网络 (GAN)。使用成员披露指标评估隐私风险。对于顺序合成，在组合至少十个合成数据集后调整模型参数可提供高决策和估计一致性、低标准化差异以及高置信区间重叠、低偏差、置信区间具有名义覆盖范围以及接近名义水平的功效。放大只有微小的好处。没有应用组合规则的单个合成数据集的置信区间覆盖范围是错误的，并且如预期的那样，当放大使用时，统计功效会被人为夸大。在多个数据集上，顺序合成的表现明显优于 GAN。对于所有数据集和模型，成员披露风险都很低。对于可重复的结果，完全合成数据的统计分析应该基于至少十个与原始数据大小相同的生成数据集，并且分析结果是组合的。不应用组合规则的合成数据的分析结果可能会产生误导。可重复性结果取决于所使用的生成模型类型，我们的研究表明，顺序合成对于常见的健康研究工作负载具有良好的可重复性特征。
