Image Optimisation and Perception Group (MIOPeG), Discipline of Clinical Imaging, Faculty of Medicine and Health, University of Sydney, Sydney, NSW, Australia.
Department of Computer Science, The University of Hong Kong, Pokfulam, Hong Kong.
PLoS One. 2023 Apr 25;18(4):e0284605. doi: 10.1371/journal.pone.0284605. eCollection 2023.
Previous studies showed that radiologists can detect the gist of an abnormality in a mammogram based on a half-second image presentation through global processing of screening mammograms. This study investigated the intra- and inter-observer reliability of the radiologists' initial impressions about the abnormality (or "gist signal"). It also examined if a subset of radiologists produced more reliable and accurate gist signals. Thirty-nine radiologists provided their initial impressions on two separate occasions, viewing each mammogram for half a second each time. The intra-class correlation (ICC) values showed poor to moderate intra-reader reliability. Only 13 radiologists had an ICC of 0.6 or above, which is considered the minimum standard for reliability, and only three radiologists had an ICC exceeding 0.7. The median value for the weighted Cohen's Kappa was 0.478 (interquartile range = 0.419-0.555). The Mann-Whitney U-test showed that the "Gist Experts", defined as those who outperformed others, had significantly higher ICC values (p = 0.002) and weighted Cohen's Kappa scores (p = 0.026). However, even for these experts, the intra-radiologist agreements were not strong, as an ICC of at least 0.75 indicates good reliability and the signal from none of the readers reached this level of reliability as determined by ICC values. The inter-reader reliability of the gist signal was poor, with an ICC score of 0.31 (CI = 0.26-0.37). The Fleiss Kappa score of 0.106 (CI = 0.105-0.106), indicating only slight inter-reader agreement, confirms the findings from the ICC analysis. The intra- and inter-reader reliability analysis showed that the radiologists' initial impressions are not reliable signals. In particular, the absence of an abnormal gist does not reliably signal a normal case, so radiologists should keep searching. This highlights the importance of "discovery scanning," or coarse screening to detect potential targets before ending the visual search.
先前的研究表明，放射科医生可以通过对筛查性乳房 X 光片进行全局处理，根据半秒的图像呈现来检测乳房 X 光片中异常的要点（或“要点信号”）。本研究调查了放射科医生对异常初始印象（或“要点信号”）的观察者内和观察者间可靠性。它还检查了是否有一部分放射科医生产生了更可靠和准确的要点信号。39 名放射科医生在两次不同的时间提供了他们的初步印象，每次观看每个乳房 X 光片半秒。组内相关系数（ICC）值显示出读者内可靠性从差到中等。只有 13 名放射科医生的 ICC 值在 0.6 或以上，这被认为是可靠性的最低标准，只有 3 名放射科医生的 ICC 值超过 0.7。加权 Cohen's Kappa 的中位数为 0.478（四分位距=0.419-0.555）。Mann-Whitney U 检验显示，“要点专家”（定义为表现优于其他人的专家）的 ICC 值明显更高（p=0.002）和加权 Cohen's Kappa 评分（p=0.026）。然而，即使对于这些专家，放射科医生内的一致性也不强，因为 ICC 值至少为 0.75 表示可靠性良好，并且没有读者的信号达到 ICC 值确定的可靠性水平。要点信号的读者间可靠性很差，ICC 评分为 0.31（CI=0.26-0.37）。Fleiss Kappa 评分为 0.106（CI=0.105-0.106），表明只有轻微的读者间一致性，这证实了 ICC 分析的结果。读者内和读者间可靠性分析表明，放射科医生的初始印象不是可靠的信号。特别是，没有异常要点并不可靠地表示正常情况，因此放射科医生应该继续搜索。这突出了“发现扫描”（或粗筛）的重要性，即在结束视觉搜索之前，通过粗筛来检测潜在目标。
