Department of Ophthalmology, Mbarara University of Science and Technology, Mbarara, Uganda.
Massachusetts General Hospital Center for Global Health, Department of Medicine, Harvard Medical School, Boston, MA, United States.
JMIR Form Res. 2024 Sep 18;8:e59914. doi: 10.2196/59914.
Labeling color fundus photos (CFP) is an important step in the development of artificial intelligence screening algorithms for the detection of diabetic retinopathy (DR). Most studies use the International Classification of Diabetic Retinopathy (ICDR) to assign labels to CFP, plus the presence or absence of macular edema (ME). Images can be grouped as referrable or nonreferrable according to these classifications. There is little guidance in the literature about how to collect and use metadata as a part of the CFP labeling process.
This study aimed to improve the quality of the Multimodal Database of Retinal Images in Africa (MoDRIA) by determining whether the availability of metadata during the image labeling process influences the accuracy, sensitivity, and specificity of image labels. MoDRIA was developed as one of the inaugural research projects of the Mbarara University Data Science Research Hub, part of the Data Science for Health Discovery and Innovation in Africa (DS-I Africa) initiative.
This is a crossover assessment with 2 groups and 2 phases. Each group had 10 randomly assigned labelers who provided an ICDR score and the presence or absence of ME for each of the 50 CFP in a test image with and without metadata including blood pressure, visual acuity, glucose, and medical history. Specificity and sensitivity of referable retinopathy were based on ICDR scores, and ME was calculated using a 2-sided t test. Comparison of sensitivity and specificity for ICDR scores and ME with and without metadata for each participant was calculated using the Wilcoxon signed rank test. Statistical significance was set at P<.05.
The sensitivity for identifying referrable DR with metadata was 92.8% (95% CI 87.6-98.0) compared with 93.3% (95% CI 87.6-98.9) without metadata, and the specificity was 84.9% (95% CI 75.1-94.6) with metadata compared with 88.2% (95% CI 79.5-96.8) without metadata. The sensitivity for identifying the presence of ME was 64.3% (95% CI 57.6-71.0) with metadata, compared with 63.1% (95% CI 53.4-73.0) without metadata, and the specificity was 86.5% (95% CI 81.4-91.5) with metadata compared with 87.7% (95% CI 83.9-91.5) without metadata. The sensitivity and specificity of the ICDR score and the presence or absence of ME were calculated for each labeler with and without metadata. No findings were statistically significant.
The sensitivity and specificity scores for the detection of referrable DR were slightly better without metadata, but the difference was not statistically significant. We cannot make definitive conclusions about the impact of metadata on the sensitivity and specificity of image labels in our study. Given the importance of metadata in clinical situations, we believe that metadata may benefit labeling quality. A more rigorous study to determine the sensitivity and specificity of CFP labels with and without metadata is recommended.
对眼底彩照（CFP）进行标注是开发用于检测糖尿病视网膜病变（DR）的人工智能筛查算法的重要步骤。大多数研究使用国际糖尿病视网膜病变分类（ICDR）对 CFP 进行标注，并加上是否存在黄斑水肿（ME）。根据这些分类，图像可以被归类为可转诊或不可转诊。文献中几乎没有关于如何收集和使用元数据作为 CFP 标注过程的一部分的指导。
本研究旨在通过确定在图像标注过程中元数据的可用性是否会影响图像标签的准确性、灵敏度和特异性，来提高非洲视网膜图像多模态数据库（MoDRIA）的质量。MoDRIA 是 Mbarara 大学数据科学研究中心的首批研究项目之一，该中心是非洲数据科学促进健康发现与创新（DS-I Africa）倡议的一部分。
这是一项具有 2 组和 2 个阶段的交叉评估。每组有 10 名随机分配的标注者，他们对 50 张测试图像中的每一张进行 ICDR 评分和 ME 的有无判断，这些图像有和没有包括血压、视力、血糖和病史在内的元数据。根据 ICDR 评分确定可转诊视网膜病变的特异性和灵敏度，使用双侧 t 检验计算 ME。使用 Wilcoxon 符号秩检验计算每组参与者在有和没有元数据的情况下对 ICDR 评分和 ME 的灵敏度和特异性的比较。P<.05 为统计学显著性。
有元数据时，识别可转诊 DR 的灵敏度为 92.8%（95%CI 87.6-98.0），而没有元数据时为 93.3%（95%CI 87.6-98.9），特异性有元数据时为 84.9%（95%CI 75.1-94.6），而没有元数据时为 88.2%（95%CI 79.5-96.8）。有元数据时，识别 ME 存在的灵敏度为 64.3%（95%CI 57.6-71.0），而没有元数据时为 63.1%（95%CI 53.4-73.0），特异性有元数据时为 86.5%（95%CI 81.4-91.5），而没有元数据时为 87.7%（95%CI 83.9-91.5）。我们对有和没有元数据的情况下每位标注者的 ICDR 评分和 ME 有无的灵敏度和特异性进行了计算。没有发现具有统计学意义的结果。
在没有元数据的情况下，可转诊 DR 的检测灵敏度和特异性得分略高，但差异无统计学意义。我们不能对元数据对 CFP 标签灵敏度和特异性的影响做出明确的结论。鉴于元数据在临床情况下的重要性，我们认为元数据可能会有益于标注质量。建议进行一项更严格的研究，以确定有和没有元数据的 CFP 标签的灵敏度和特异性。
