Peeters Dré, Venkadesh Kiran V, Dinnessen Renate, Saghir Zaigham, Scholten Ernst T, Vliegenthart Rozemarijn, Prokop Mathias, Jacobs Colin
Diagnostic Imaging Analysis Group, Medical Imaging Department, Radboud University Medical Center, Geert Grooteplein Zuid 10, 6525 GA, Nijmegen, the Netherlands.
Diagnostic Imaging Analysis Group, Medical Imaging Department, Radboud University Medical Center, Geert Grooteplein Zuid 10, 6525 GA, Nijmegen, the Netherlands.
Comput Biol Med. 2025 Mar;186:109633. doi: 10.1016/j.compbiomed.2024.109633. Epub 2024 Dec 30.
Artificial Intelligence (AI) models may fail or suffer from reduced performance when applied to unseen data that differs from the training data distribution, referred to as dataset shift. Automatic detection of out-of-distribution (OOD) data contributes to safe and reliable clinical implementation of AI models. In this study, we propose a recognized OOD detection method that utilizes the Mahalanobis distance (MD) and compare its performance to widely known classical methods. The MD measures the similarity between features of an unseen sample and the distribution of development samples features of intermediate model layers. We integrate our proposed method in an existing deep learning (DL) model for lung nodule malignancy risk estimation on chest CT and validate it across four dataset shifts known to reduce AI model performance. The results show that our proposed method outperforms the classical methods and can effectively detect near- and far-OOD samples across all datasets with different data distribution shifts. Additionally, we demonstrate that our proposed method can seamlessly incorporate additional In-distribution (ID) data while maintaining the ability to accurately differentiate between the remaining OOD cases. Lastly, we searched for the optimal OOD threshold in the OOD dataset where the performance of the DL model stays reliable, however no decline in DL performance was revealed as the OOD score increased.
当人工智能（AI）模型应用于与训练数据分布不同的未见数据时，可能会失败或性能下降，这种情况被称为数据集偏移。自动检测分布外（OOD）数据有助于AI模型在临床中安全可靠地应用。在本研究中，我们提出一种利用马氏距离（MD）的公认OOD检测方法，并将其性能与广为人知的经典方法进行比较。MD用于衡量未见样本的特征与中间模型层的开发样本特征分布之间的相似性。我们将所提出的方法集成到现有的深度学习（DL）模型中，用于胸部CT上肺结节恶性风险评估，并在已知会降低AI模型性能的四种数据集偏移情况下对其进行验证。结果表明，我们提出的方法优于经典方法，能够有效检测所有具有不同数据分布偏移的数据集上的近OOD样本和远OOD样本。此外，我们证明了所提出的方法可以无缝合并额外的分布内（ID）数据，同时保持准确区分其余OOD病例的能力。最后，我们在DL模型性能保持可靠的OOD数据集中搜索最佳OOD阈值，然而随着OOD分数的增加，DL性能并未下降。
