IEEE Trans Image Process. 2023;32:3493-3506. doi: 10.1109/TIP.2023.3286256. Epub 2023 Jun 23.
Intra prediction is a crucial part of video compression, which utilizes local information in images to eliminate spatial redundancy. As the state-of-the-art video coding standard, Versatile Video Coding (H.266/VVC) employs multiple directional prediction modes in intra prediction to find the texture trend of local areas. Then the prediction is made based on reference samples in the selected direction. Recently, neural network-based intra prediction has achieved great success. Deep network models are trained and applied to assist the HEVC and VVC intra modes. In this paper, we propose a novel tree-structured data clustering-driven neural network (dubbed TreeNet) for intra prediction, which builds the networks and clusters the training data in a tree-structured manner. Specifically, in each network split and training process of TreeNet, every parent network on a leaf node is split into two child networks by adding or subtracting Gaussian random noise. Then data clustering-driven training is applied to train the two derived child networks using the clustered training data of their parent. On the one hand, the networks at the same level in TreeNet are trained with non-overlapping clustered datasets, and thus they can learn different prediction abilities. On the other hand, the networks at different levels are trained with hierarchically clustered datasets, and thus they will have different generalization abilities. TreeNet is integrated into VVC to assist or replace intra prediction modes to test its performance. In addition, a fast termination strategy is proposed to accelerate the search of TreeNet. The experimental results demonstrate that when TreeNet is used to assist the VVC Intra modes, TreeNet with depth = 3 can bring an average of 3.78% bitrate saving (up to 8.12%) over VTM-17.0. If TreeNet with the same depth replaces all VVC intra modes, an average of 1.59% bitrate saving can be reached.
帧内预测是视频压缩的一个重要组成部分，它利用图像中的局部信息来消除空间冗余。作为最新的视频编码标准，多功能视频编码（H.266/VVC）在帧内预测中采用了多种方向预测模式，以找到局部区域的纹理趋势。然后基于所选方向的参考样本进行预测。最近，基于神经网络的帧内预测取得了巨大的成功。深度网络模型被训练并应用于协助 HEVC 和 VVC 帧内模式。在本文中，我们提出了一种新颖的基于树状结构数据聚类的神经网络（称为 TreeNet）用于帧内预测，该方法以树状结构的方式构建网络并对训练数据进行聚类。具体来说，在 TreeNet 的每个网络分裂和训练过程中，叶节点上的每个父网络通过添加或减去高斯随机噪声被分裂成两个子网络。然后，使用聚类后的训练数据对两个衍生的子网络进行数据聚类驱动训练。一方面，TreeNet 中的同层网络使用非重叠聚类数据集进行训练，因此它们可以学习不同的预测能力。另一方面，不同层的网络使用层次聚类数据集进行训练，因此它们将具有不同的泛化能力。TreeNet 被集成到 VVC 中，以辅助或替代帧内预测模式来测试其性能。此外，还提出了一种快速终止策略来加速 TreeNet 的搜索。实验结果表明，当 TreeNet 用于辅助 VVC 帧内模式时，深度=3 的 TreeNet 可以比 VTM-17.0 平均节省 3.78%的比特率（最高可达 8.12%）。如果使用相同深度的 TreeNet 替换所有 VVC 帧内模式，则可以平均节省 1.59%的比特率。
