Bao Zhiqiang, Zhu Di, Du Leying, Li Yang
School of Communication and Information Engineering, Xi'an University of Posts & Telecommunications, Xian, 710121, China.
School of Artificial Intelligence, Xi'an University of Posts & Telecommunications, Xian, 710121, China.
Sci Rep. 2025 Apr 16;15(1):13197. doi: 10.1038/s41598-025-97699-5.
Within the research scope of knowledge distillation, contrastive representation distillation has achieved remarkable research results by introducing the Contrastive Representation Distillation Loss. However, previous research has relatively paid scant attention to the influence of factors at the input sample level. We observe that the existence of a large number of negative sample pairs in the knowledge transfer process leads to the issue of information redundancy. To mitigate this issue, we propose a representation normalization method and apply it to contrastive representation distillation. The method aims to reduce the information redundancy caused by negative sample pairs. Meanwhile, drawing on the idea of the Triplet Loss function in contrastive learning, we constructed a loss function and integrated it into the Contrastive Representation Distillation Loss to form the Contrast Enhanced Representation Normalization Distillation Loss. This new loss function aims to enhance the similarity between positive sample pairs and increase the distance between negative sample pairs. The experimental results demonstrate that the Contrast Enhanced Representation Normalization Distillation algorithm outperforms the Contrastive Representation Distillation algorithm on the CIFAR100 and ImageNet datasets, and shows remarkable performance compared with other state-of-the-art knowledge distillation methods.This not only enables the deployment of models on resource-constrained devices, but also demonstrates extensive potential application values in tasks such as image segmentation, providing strong support for related research and practical applications.
在知识蒸馏的研究范围内，对比表示蒸馏通过引入对比表示蒸馏损失取得了显著的研究成果。然而，以往的研究相对较少关注输入样本层面因素的影响。我们观察到，知识转移过程中大量负样本对的存在导致了信息冗余问题。为缓解这一问题，我们提出了一种表示归一化方法，并将其应用于对比表示蒸馏。该方法旨在减少负样本对造成的信息冗余。同时，借鉴对比学习中三元组损失函数的思想，我们构建了一个损失函数，并将其集成到对比表示蒸馏损失中，形成对比增强表示归一化蒸馏损失。这个新的损失函数旨在增强正样本对之间的相似度，并增大负样本对之间的距离。实验结果表明，对比增强表示归一化蒸馏算法在CIFAR100和ImageNet数据集上优于对比表示蒸馏算法，并且与其他先进的知识蒸馏方法相比表现出色。这不仅使得模型能够在资源受限的设备上部署，还在图像分割等任务中展现出广泛的潜在应用价值，为相关研究和实际应用提供了有力支持。
