IEEE Trans Image Process. 2022;31:4405-4416. doi: 10.1109/TIP.2022.3180208. Epub 2022 Jul 1.
While humans can effortlessly transform complex visual scenes into simple words and the other way around by leveraging their high-level understanding of the content, conventional or the more recent learned image compression codecs do not seem to utilize the semantic meanings of visual content to their full potential. Moreover, they focus mostly on rate-distortion and tend to underperform in perception quality especially in low bitrate regime, and often disregard the performance of downstream computer vision algorithms, which is a fast-growing consumer group of compressed images in addition to human viewers. In this paper, we (1) present a generic framework that can enable any image codec to leverage high-level semantics and (2) study the joint optimization of perception quality and distortion. Our idea is that given any codec, we utilize high-level semantics to augment the low-level visual features extracted by it and produce essentially a new, semantic-aware codec. We propose a three-phase training scheme that teaches semantic-aware codecs to leverage the power of semantic to jointly optimize rate-perception-distortion (R-PD) performance. As an additional benefit, semantic-aware codecs also boost the performance of downstream computer vision algorithms. To validate our claim, we perform extensive empirical evaluations and provide both quantitative and qualitative results.
虽然人类可以毫不费力地通过利用其对内容的高级理解，将复杂的视觉场景转换为简单的文字，反之亦然，但传统的或最近的学习图像压缩编解码器似乎并没有充分利用视觉内容的语义含义。此外，它们主要侧重于率失真，并且在感知质量方面表现不佳，特别是在低比特率范围内，并且经常忽略下游计算机视觉算法的性能，除了人类观察者之外，这些算法是压缩图像的快速增长的消费群体。在本文中，我们 (1) 提出了一个通用框架，使任何图像编解码器都能够利用高级语义，(2) 研究了感知质量和失真的联合优化。我们的想法是，给定任何编解码器，我们利用高级语义来增强它提取的低级视觉特征，并生成一个本质上是新的、具有语义意识的编解码器。我们提出了一个三阶段的训练方案，该方案教导语义感知编解码器利用语义的力量来共同优化率感知失真 (R-PD) 性能。作为额外的好处，语义感知编解码器还可以提高下游计算机视觉算法的性能。为了验证我们的说法，我们进行了广泛的实证评估，并提供了定量和定性的结果。
