National Institute for Health Research Biomedical Research Centre at Moorfields Eye Hospital & University College London Institute of Ophthalmology, London, United Kingdom.
PLoS One. 2013 Aug 21;8(8):e71154. doi: 10.1371/journal.pone.0071154. eCollection 2013.
Crowdsourcing is the process of outsourcing numerous tasks to many untrained individuals. Our aim was to assess the performance and repeatability of crowdsourcing for the classification of retinal fundus photography.
One hundred retinal fundus photograph images with pre-determined disease criteria were selected by experts from a large cohort study. After reading brief instructions and an example classification, we requested that knowledge workers (KWs) from a crowdsourcing platform classified each image as normal or abnormal with grades of severity. Each image was classified 20 times by different KWs. Four study designs were examined to assess the effect of varying incentive and KW experience in classification accuracy. All study designs were conducted twice to examine repeatability. Performance was assessed by comparing the sensitivity, specificity and area under the receiver operating characteristic curve (AUC).
Without restriction on eligible participants, two thousand classifications of 100 images were received in under 24 hours at minimal cost. In trial 1 all study designs had an AUC (95%CI) of 0.701(0.680-0.721) or greater for classification of normal/abnormal. In trial 1, the highest AUC (95%CI) for normal/abnormal classification was 0.757 (0.738-0.776) for KWs with moderate experience. Comparable results were observed in trial 2. In trial 1, between 64-86% of any abnormal image was correctly classified by over half of all KWs. In trial 2, this ranged between 74-97%. Sensitivity was ≥ 96% for normal versus severely abnormal detections across all trials. Sensitivity for normal versus mildly abnormal varied between 61-79% across trials.
With minimal training, crowdsourcing represents an accurate, rapid and cost-effective method of retinal image analysis which demonstrates good repeatability. Larger studies with more comprehensive participant training are needed to explore the utility of this compelling technique in large scale medical image analysis.
众包是将许多任务外包给许多未经培训的个人的过程。我们的目的是评估众包在视网膜眼底照相分类中的性能和可重复性。
从一项大型队列研究中，由专家选择了 100 张具有预定疾病标准的视网膜眼底照片。在阅读了简短的说明和一个分类示例后，我们要求众包平台上的知识工作者（KW）将每张图像分类为正常或异常，并分级严重程度。每张图像由不同的 KWs 分类 20 次。检查了四种研究设计，以评估不同激励和 KW 经验对分类准确性的影响。所有研究设计均进行了两次以检查可重复性。通过比较敏感性、特异性和接收者操作特征曲线下的面积（AUC）来评估性能。
在不受合格参与者限制的情况下，在不到 24 小时的时间内以最低成本收到了 2000 次对 100 张图像的分类。在试验 1 中，所有研究设计的正常/异常分类的 AUC（95%CI）均为 0.701（0.680-0.721）或更高。在试验 1 中，具有中等经验的 KWs 的正常/异常分类的 AUC（95%CI）最高为 0.757（0.738-0.776）。试验 2 中观察到类似的结果。在试验 1 中，超过一半的 KWs 正确分类了任何异常图像的 64-86%。在试验 2 中，这一比例在 74-97%之间。所有试验中，正常与严重异常检测的敏感性均≥96%。正常与轻度异常检测的敏感性在试验之间变化 61-79%。
经过最少的培训，众包代表了一种准确、快速且具有成本效益的视网膜图像分析方法，具有良好的可重复性。需要进行更大规模的研究，并对参与者进行更全面的培训，以探索这种引人注目的技术在大规模医学图像分析中的应用。
