Adaptive multi-layer filters incorporated with Volterra filters for impairment compensation including transmitter and receiver nonlinearity.

We propose a receiver-side signal processing to compensate for nonlinearity that occurs in transmitter (Tx) and receiver (Rx) components of coherent optical fiber transmission systems. Nonlinear effects in transmission systems are not mutually commutative with any linear effects in general. Considering the order in which all the relevant impairments occur, we adopt a multi-layer (ML) filter architecture. The ML filters consist of strictly-linear and widely-linear filter layers to compensate for relevant linear impairments that occur in a transmission system and two Volterra filter layers to compensate for Rx and Tx nonlinearity. The coefficients of the ML filters including Volterra filter layers are adaptively controlled by using a gradient calculation with back propagation, which is similar to that used in the learning of neural networks, from the last layer and stochastic gradient descent to minimize a loss function that is composed of the last layer outputs. We evaluated the compensation performance of Tx and Rx nonlinearity using the proposed adaptive ML filters including Volterra filter layers both in simulations and experiments of the transmission of a 23 Gbaud polarization-division-multiplexed 64-quadrature amplitude modulation signal over a 100-km single-mode-fiber span. The results demonstrated that the Volterra filter layers in the ML filter architecture could compensate for the nonlinearity that occurs in Tx and Rx simultaneously and effectively even when other impairments such as chromatic dispersion coexist.