Institute of High Performance Computing (IHPC), Agency for Science, Technology and Research (A*STAR), 1 Fusionopolis Way, #16-16 Connexis, Singapore, 138632, Republic of Singapore.
Joint Shantou International Eye Center, Shantou University and the Chinese University of Hong Kong, 515041, Shantou, Guangdong, China.
Nat Commun. 2023 Oct 24;14(1):6757. doi: 10.1038/s41467-023-42444-7.
Failure to recognize samples from the classes unseen during training is a major limitation of artificial intelligence in the real-world implementation for recognition and classification of retinal anomalies. We establish an uncertainty-inspired open set (UIOS) model, which is trained with fundus images of 9 retinal conditions. Besides assessing the probability of each category, UIOS also calculates an uncertainty score to express its confidence. Our UIOS model with thresholding strategy achieves an F1 score of 99.55%, 97.01% and 91.91% for the internal testing set, external target categories (TC)-JSIEC dataset and TC-unseen testing set, respectively, compared to the F1 score of 92.20%, 80.69% and 64.74% by the standard AI model. Furthermore, UIOS correctly predicts high uncertainty scores, which would prompt the need for a manual check in the datasets of non-target categories retinal diseases, low-quality fundus images, and non-fundus images. UIOS provides a robust method for real-world screening of retinal anomalies.
未能识别训练中未见过的样本是人工智能在实际应用中对视网膜异常进行识别和分类的主要限制。我们建立了一个基于不确定性的开放集 (UIOS) 模型，该模型使用 9 种视网膜疾病的眼底图像进行训练。除了评估每个类别的概率外，UIOS 还计算不确定性得分来表达其置信度。与标准 AI 模型相比，我们的 UIOS 模型与阈值策略在内部测试集、外部目标类别 (TC)-JSIEC 数据集和 TC-未见测试集上的 F1 分数分别为 99.55%、97.01%和 91.91%，而标准 AI 模型的 F1 分数分别为 92.20%、80.69%和 64.74%。此外，UIOS 正确预测了高不确定性得分，这将促使需要在非目标类别视网膜疾病、低质量眼底图像和非眼底图像的数据集进行手动检查。UIOS 为视网膜异常的实际筛选提供了一种稳健的方法。
