Imaging Biomarkers and Computer-Aided Diagnosis Laboratory, Radiology and Imaging Sciences, National Institutes of Health Clinical Center, Bethesda, MD 20892-1182, United States.
Med Image Anal. 2012 Aug;16(6):1280-92. doi: 10.1016/j.media.2012.04.007. Epub 2012 May 3.
Computer-aided detection (CAD) systems have been shown to improve the diagnostic performance of CT colonography (CTC) in the detection of premalignant colorectal polyps. Despite the improvement, the overall system is not optimal. CAD annotations on true lesions are incorrectly dismissed, and false positives are misinterpreted as true polyps. Here, we conduct an observer performance study utilizing distributed human intelligence in the form of anonymous knowledge workers (KWs) to investigate human performance in classifying polyp candidates under different presentation strategies. We evaluated 600 polyp candidates from 50 patients, each case having at least one polyp ≥6 mm, from a large database of CTC studies. Each polyp candidate was labeled independently as a true or false polyp by 20 KWs and an expert radiologist. We asked each labeler to determine whether the candidate was a true polyp after looking at a single 3D-rendered image of the candidate and after watching a video fly-around of the candidate. We found that distributed human intelligence improved significantly when presented with the additional information in the video fly-around. We noted that performance degraded with increasing interpretation time and increasing difficulty, but distributed human intelligence performed better than our CAD classifier for "easy" and "moderate" polyp candidates. Further, we observed numerous parallels between the expert radiologist and the KWs. Both showed similar improvement in classification moving from single-image to video interpretation. Additionally, difficulty estimates obtained from the KWs using an expectation maximization algorithm correlated well with the difficulty rating assigned by the expert radiologist. Our results suggest that distributed human intelligence is a powerful tool that will aid in the development of CAD for CTC.
计算机辅助检测（CAD）系统已被证明可提高 CT 结肠成像（CTC）检测癌前结直肠息肉的诊断性能。尽管有所改善，但整个系统并不理想。CAD 对真正病变的注释被错误地忽略，而假阳性则被错误地解释为真正的息肉。在这里，我们进行了一项观察者性能研究，利用分布式人类智能（以匿名知识工作者（KW）的形式）来研究在不同呈现策略下分类息肉候选物的人类性能。我们评估了来自大型 CTC 研究数据库的 50 名患者中至少有一个≥6mm 息肉的 600 个息肉候选物。每个息肉候选物都由 20 名 KWs 和一名专家放射科医生独立标记为真或假息肉。我们要求每个标记者在查看候选物的单个 3D 渲染图像后，并在观看候选物的视频环绕后，确定候选物是否为真息肉。我们发现，当提供视频环绕中的附加信息时，分布式人类智能显著提高。我们注意到，随着解释时间的增加和难度的增加，性能会下降，但分布式人类智能在“简单”和“中等”息肉候选物的表现优于我们的 CAD 分类器。此外，我们观察到专家放射科医生和 KWs 之间存在许多相似之处。两者都从单一图像到视频解释的分类中表现出类似的改善。此外，使用期望最大化算法从 KWs 获得的难度估计与专家放射科医生分配的难度评分相关良好。我们的结果表明，分布式人类智能是一种强大的工具，将有助于开发 CTC 的 CAD。
