Continual learning across population cohorts with distribution shift: insights from multi-cohort metabolic syndrome identification.

OBJECTIVE

This study aims to tackle the critical challenge of adapting deep learning (DL) models for deployment in real-world healthcare settings, specifically focusing on catastrophic forgetting due to distribution shifts between hospital and non-hospital environments. Metabolic syndrome (MetS) is susceptible to misdiagnosis by DL models due to distribution shifts. This work demonstrates the potential of continual learning (CL) to enhance model performance in MetS identification across diverse settings.

MATERIALS AND METHODS

We utilized 3 healthcare datasets from 2 distinct settings: Medical Information Mart for Intensive Care (MIMIC; from a hospital setting), National Health and Nutrition Examination Survey (NHANES), and a proprietary dataset (both from non-hospital settings). We proposed a robust MetS identification pipeline with CL strategies and evaluated its effectiveness in mitigating catastrophic forgetting while maintaining high predictive performance under distribution shift.

RESULTS

The CL method outperformed the control (sequential training without any strategies) method. The CL method reached a cumulative area under the ROC curve (AUROC) of 0.85 and area under the precision-recall curve of 0.65 on the combined test set. Moreover, training order proved critical: models trained from hospital to non-hospital settings achieved a 7.6% improvement in AUROC, increasing from 0.79 to 0.85, compared to the reverse order.

DISCUSSION

Our results demonstrate the potential of CL for applications across healthcare settings, particularly between hospital and non-hospital settings. We also discuss the impact of training order on the results.

CONCLUSION

The proposed CL model effectively mitigates catastrophic forgetting, enhancing the overall performance of DL models. Our results underscore the prospect of CL methods in developing medical DL models and maintaining scalability across diverse healthcare settings.