Department of Computer Engineering, Sharif University of Technology, Tehran, Iran.
PLoS One. 2024 Oct 7;19(10):e0307432. doi: 10.1371/journal.pone.0307432. eCollection 2024.
Recently, there has been growing interest in deep spectral methods for image localization and segmentation, influenced by traditional spectral segmentation approaches. These methods reframe the image decomposition process as a graph partitioning task by extracting features using self-supervised learning and utilizing the Laplacian of the affinity matrix to obtain eigensegments. However, instance segmentation has received less attention than other tasks within the context of deep spectral methods. This paper addresses that not all channels of the feature map extracted from a self-supervised backbone contain sufficient information for instance segmentation purposes. Some channels are noisy and hinder the accuracy of the task. To overcome this issue, this paper proposes two channel reduction modules, Noise Channel Reduction (NCR) and Deviation-based Channel Reduction (DCR). The NCR retains channels with lower entropy, as they are less likely to be noisy, while DCR prunes channels with low standard deviation, as they lack sufficient information for effective instance segmentation. Furthermore, the paper demonstrates that the dot product, commonly used in deep spectral methods, is not suitable for instance segmentation due to its sensitivity to feature map values, potentially leading to incorrect instance segments. A novel similarity metric called Bray-curtis over Chebyshev (BoC) is proposed to address this issue. This metric considers the distribution of features in addition to their values, providing a more robust similarity measure for instance segmentation. Quantitative and qualitative results on the Youtube-VIS 2019 and OVIS datasets highlight the improvements achieved by the proposed channel reduction methods and using BoC instead of the conventional dot product for creating the affinity matrix. These improvements regarding mean Intersection over Union (mIoU) and extracted instance segments are observed, demonstrating enhanced instance segmentation performance. The code is available on: https://github.com/farnooshar/SpecUnIIS.
最近，由于传统的谱分割方法的影响，人们对用于图像定位和分割的深度光谱方法越来越感兴趣。这些方法通过使用自监督学习提取特征，并利用亲和矩阵的拉普拉斯算子获得特征段，将图像分解过程重新表述为图划分任务。然而，在深度光谱方法的背景下，实例分割比其他任务受到的关注较少。本文认为，并非自监督主干提取的特征图的所有通道都包含足够的信息用于实例分割。一些通道是嘈杂的，会阻碍任务的准确性。为了解决这个问题，本文提出了两个通道减少模块，噪声通道减少（NCR）和基于偏差的通道减少（DCR）。NCR 保留具有较低熵的通道，因为它们不太可能是嘈杂的，而 DCR 修剪具有低标准差的通道，因为它们缺乏有效实例分割所需的足够信息。此外，本文还证明了，在深度光谱方法中常用的点积由于其对特征图值的敏感性，不适合用于实例分割，可能导致错误的实例分割。提出了一种新的相似性度量方法，称为切比雪夫上的布雷-柯蒂斯（BoC），以解决这个问题。该度量方法不仅考虑了特征值，还考虑了特征的分布，为实例分割提供了更稳健的相似性度量。在 Youtube-VIS 2019 和 OVIS 数据集上的定量和定性结果突出了所提出的通道减少方法和使用 BoC 代替传统的点积来创建亲和矩阵所带来的改进。观察到关于平均交并比（mIoU）和提取的实例段的改进，表明实例分割性能得到了增强。代码可在：https://github.com/farnooshar/SpecUnIIS 上获得。
