Department of Computer Science and Engineering, University of Minnesota, Minneapolis, Minnesota, USA.
Department of Computer Science, Emory University, Atlanta, Georgia, USA.
J Am Med Inform Assoc. 2022 Dec 13;30(1):54-63. doi: 10.1093/jamia/ocac188.
Federated learning (FL) allows multiple distributed data holders to collaboratively learn a shared model without data sharing. However, individual health system data are heterogeneous. "Personalized" FL variations have been developed to counter data heterogeneity, but few have been evaluated using real-world healthcare data. The purpose of this study is to investigate the performance of a single-site versus a 3-client federated model using a previously described Coronavirus Disease 19 (COVID-19) diagnostic model. Additionally, to investigate the effect of system heterogeneity, we evaluate the performance of 4 FL variations.
We leverage a FL healthcare collaborative including data from 5 international healthcare systems (US and Europe) encompassing 42 hospitals. We implemented a COVID-19 computer vision diagnosis system using the Federated Averaging (FedAvg) algorithm implemented on Clara Train SDK 4.0. To study the effect of data heterogeneity, training data was pooled from 3 systems locally and federation was simulated. We compared a centralized/pooled model, versus FedAvg, and 3 personalized FL variations (FedProx, FedBN, and FedAMP).
We observed comparable model performance with respect to internal validation (local model: AUROC 0.94 vs FedAvg: 0.95, P = .5) and improved model generalizability with the FedAvg model (P < .05). When investigating the effects of model heterogeneity, we observed poor performance with FedAvg on internal validation as compared to personalized FL algorithms. FedAvg did have improved generalizability compared to personalized FL algorithms. On average, FedBN had the best rank performance on internal and external validation.
FedAvg can significantly improve the generalization of the model compared to other personalization FL algorithms; however, at the cost of poor internal validity. Personalized FL may offer an opportunity to develop both internal and externally validated algorithms.
联邦学习（FL）允许多个分布式数据持有者在不共享数据的情况下协作学习共享模型。然而，个体健康系统数据具有异质性。已经开发了“个性化”的 FL 变体来应对数据异质性，但很少有使用真实医疗保健数据进行评估的变体。本研究的目的是使用以前描述的 2019 冠状病毒病（COVID-19）诊断模型，研究单个站点与 3 个客户端联邦模型的性能。此外，为了研究系统异质性的影响，我们评估了 4 种 FL 变体的性能。
我们利用包括来自 5 个国际医疗保健系统（美国和欧洲）的 42 家医院的数据的 FL 医疗保健协作，该系统包括使用 Clara Train SDK 4.0 上的联邦平均（FedAvg）算法实现的 COVID-19 计算机视觉诊断系统。为了研究数据异质性的影响，我们在本地从 3 个系统中汇集了训练数据并模拟了联邦学习。我们比较了集中式/汇集模型与 FedAvg 以及 3 种个性化 FL 变体（FedProx、FedBN 和 FedAMP）。
我们观察到内部验证方面的模型性能相当（本地模型：AUROC 0.94 与 FedAvg：0.95，P=0.5），FedAvg 模型的模型通用性得到了提高（P<0.05）。在研究模型异质性的影响时，我们观察到 FedAvg 在内部验证中的性能与个性化 FL 算法相比较差。FedAvg 与个性化 FL 算法相比确实具有更好的通用性。平均而言，FedBN 在内部和外部验证中的排名性能最好。
FedAvg 与其他个性化 FL 算法相比，可以显著提高模型的通用性；但是，这是以内部有效性较差为代价的。个性化 FL 可能为开发内部和外部验证算法提供机会。
