Li Siqi, Ning Yilin, Ong Marcus Eng Hock, Chakraborty Bibhas, Hong Chuan, Xie Feng, Yuan Han, Liu Mingxuan, Buckland Daniel M, Chen Yong, Liu Nan
Centre for Quantitative Medicine, Duke-NUS Medical School, Singapore, Singapore.
Programme in Health Services and Systems Research, Duke-NUS Medical School, Singapore, Singapore; Health Services Research Centre, Singapore Health Services, Singapore, Singapore; Department of Emergency Medicine, Singapore General Hospital, Singapore, Singapore.
J Biomed Inform. 2023 Oct;146:104485. doi: 10.1016/j.jbi.2023.104485. Epub 2023 Sep 1.
We propose FedScore, a privacy-preserving federated learning framework for scoring system generation across multiple sites to facilitate cross-institutional collaborations.
The FedScore framework includes five modules: federated variable ranking, federated variable transformation, federated score derivation, federated model selection and federated model evaluation. To illustrate usage and assess FedScore's performance, we built a hypothetical global scoring system for mortality prediction within 30 days after a visit to an emergency department using 10 simulated sites divided from a tertiary hospital in Singapore. We employed a pre-existing score generator to construct 10 local scoring systems independently at each site and we also developed a scoring system using centralized data for comparison.
We compared the acquired FedScore model's performance with that of other scoring models using the receiver operating characteristic (ROC) analysis. The FedScore model achieved an average area under the curve (AUC) value of 0.763 across all sites, with a standard deviation (SD) of 0.020. We also calculated the average AUC values and SDs for each local model, and the FedScore model showed promising accuracy and stability with a high average AUC value which was closest to the one of the pooled model and SD which was lower than that of most local models.
This study demonstrates that FedScore is a privacy-preserving scoring system generator with potentially good generalizability.
我们提出了FedScore，这是一个用于跨多个站点生成评分系统的隐私保护联邦学习框架，以促进跨机构合作。
FedScore框架包括五个模块：联邦变量排序、联邦变量转换、联邦分数推导、联邦模型选择和联邦模型评估。为了说明其用法并评估FedScore的性能，我们使用从新加坡一家三级医院划分出的10个模拟站点，构建了一个用于预测急诊科就诊后30天内死亡率的假设全球评分系统。我们使用一个预先存在的分数生成器在每个站点独立构建10个本地评分系统，并且我们还使用集中数据开发了一个评分系统用于比较。
我们使用受试者工作特征（ROC）分析将获得的FedScore模型的性能与其他评分模型进行了比较。FedScore模型在所有站点的平均曲线下面积（AUC）值为0.763，标准差（SD）为0.020。我们还计算了每个本地模型的平均AUC值和标准差，FedScore模型显示出有前景的准确性和稳定性，其平均AUC值较高，最接近合并模型的AUC值，标准差低于大多数本地模型。
本研究表明FedScore是一个具有潜在良好通用性的隐私保护评分系统生成器。
