Wang Ruhua, An Senjian, Liu Wanquan, Li Ling
IEEE Trans Neural Netw Learn Syst. 2024 Jul;35(7):10167-10173. doi: 10.1109/TNNLS.2023.3238397. Epub 2024 Jul 8.
Residual blocks have been widely used in deep learning networks. However, information may be lost in residual blocks due to the relinquishment of information in rectifier linear units (ReLUs). To address this issue, invertible residual networks have been proposed recently but are generally under strict restrictions which limit their applications. In this brief, we investigate the conditions under which a residual block is invertible. A sufficient and necessary condition is presented for the invertibility of residual blocks with one layer of ReLU inside the block. In particular, for widely used residual blocks with convolutions, we show that such residual blocks are invertible under weak conditions if the convolution is implemented with certain zero-padding methods. Inverse algorithms are also proposed, and experiments are conducted to show the effectiveness of the proposed inverse algorithms and prove the correctness of the theoretical results.
残差块已在深度学习网络中得到广泛应用。然而，由于整流线性单元（ReLU）中信息的舍弃，信息可能会在残差块中丢失。为了解决这个问题，最近提出了可逆残差网络，但它们通常受到严格限制，这限制了它们的应用。在本简报中，我们研究了残差块可逆的条件。给出了块内有一层ReLU的残差块可逆性的充分必要条件。特别地，对于广泛使用的带有卷积的残差块，我们表明如果卷积采用特定的零填充方法，那么在弱条件下这种残差块是可逆的。还提出了逆算法，并进行了实验以证明所提出的逆算法的有效性，并证明理论结果的正确性。
