Institute of Biomedical Engineering and Big Data Institute, Oxford, UK; University of Oxford, Old Road Campus, Oxford, UK; Oxford NIHR Biomedical Research Centre, Oxford, UK.
Ludwig Institute for Cancer Research, Oxford, UK; University of Oxford, Old Road Campus, Oxford, UK.
Med Image Anal. 2021 Feb;68:101900. doi: 10.1016/j.media.2020.101900. Epub 2020 Nov 13.
Endoscopy is a routine imaging technique used for both diagnosis and minimally invasive surgical treatment. Artifacts such as motion blur, bubbles, specular reflections, floating objects and pixel saturation impede the visual interpretation and the automated analysis of endoscopy videos. Given the widespread use of endoscopy in different clinical applications, robust and reliable identification of such artifacts and the automated restoration of corrupted video frames is a fundamental medical imaging problem. Existing state-of-the-art methods only deal with the detection and restoration of selected artifacts. However, typically endoscopy videos contain numerous artifacts which motivates to establish a comprehensive solution. In this paper, a fully automatic framework is proposed that can: 1) detect and classify six different artifacts, 2) segment artifact instances that have indefinable shapes, 3) provide a quality score for each frame, and 4) restore partially corrupted frames. To detect and classify different artifacts, the proposed framework exploits fast, multi-scale and single stage convolution neural network detector. In addition, we use an encoder-decoder model for pixel-wise segmentation of irregular shaped artifacts. A quality score is introduced to assess video frame quality and to predict image restoration success. Generative adversarial networks with carefully chosen regularization and training strategies for discriminator-generator networks are finally used to restore corrupted frames. The detector yields the highest mean average precision (mAP) of 45.7 and 34.7, respectively for 25% and 50% IoU thresholds, and the lowest computational time of 88 ms allowing for near real-time processing. The restoration models for blind deblurring, saturation correction and inpainting demonstrate significant improvements over previous methods. On a set of 10 test videos, an average of 68.7% of video frames successfully passed the quality score (≥0.9) after applying the proposed restoration framework thereby retaining 25% more frames compared to the raw videos. The importance of artifacts detection and their restoration on improved robustness of image analysis methods is also demonstrated in this work.
内窥镜检查是一种用于诊断和微创治疗的常规成像技术。运动模糊、气泡、镜面反射、漂浮物体和像素饱和等伪影会妨碍内窥镜视频的视觉解释和自动分析。鉴于内窥镜在不同临床应用中的广泛使用，稳健可靠地识别这些伪影并自动恢复损坏的视频帧是一个基本的医学成像问题。现有的最先进方法仅处理选定伪影的检测和恢复。然而，典型的内窥镜视频包含许多伪影，这促使我们建立一个全面的解决方案。在本文中，提出了一种全自动框架，可以：1）检测和分类六种不同的伪影，2）分割具有不定形状的伪影实例，3）为每个帧提供质量评分，以及 4）恢复部分损坏的帧。为了检测和分类不同的伪影，所提出的框架利用快速、多尺度和单级卷积神经网络检测器。此外，我们使用编码器-解码器模型进行不规则形状伪影的逐像素分割。质量评分用于评估视频帧质量并预测图像恢复成功。生成对抗网络（GAN）使用精心选择的正则化和训练策略用于判别器-生成器网络，最后用于恢复损坏的帧。该检测器的平均精度（mAP）最高分别为 45.7 和 34.7，对于 25%和 50%的 IoU 阈值，计算时间最短为 88ms，允许近实时处理。盲去模糊、饱和度校正和修复的恢复模型均显著优于以前的方法。在一组 10 个测试视频上，应用所提出的恢复框架后，平均有 68.7%的视频帧成功通过质量评分（≥0.9），与原始视频相比，保留了 25%的更多帧。在这项工作中还证明了伪影检测及其恢复对于提高图像分析方法的鲁棒性的重要性。
