Luo Yi, Mesgarani Nima
IEEE/ACM Trans Audio Speech Lang Process. 2019 Aug;27(8):1256-1266. doi: 10.1109/TASLP.2019.2915167. Epub 2019 May 6.
Single-channel, speaker-independent speech separation methods have recently seen great progress. However, the accuracy, latency, and computational cost of such methods remain insufficient. The majority of the previous methods have formulated the separation problem through the time-frequency representation of the mixed signal, which has several drawbacks, including the decoupling of the phase and magnitude of the signal, the suboptimality of time-frequency representation for speech separation, and the long latency of the entire system. To address these shortcomings, we propose a fully-convolutional time-domain audio separation network (Conv-TasNet), a deep learning framework for end-to-end time-domain speech separation. Conv-TasNet uses a linear encoder to generate a representation of the speech waveform optimized for separating individual speakers. Speaker separation is achieved by applying a set of weighting functions (masks) to the encoder output. The modified encoder representations are then inverted back to the waveforms using a linear decoder. The masks are found using a temporal convolutional network (TCN) consisting of stacked 1-D dilated convolutional blocks, which allows the network to model the long-term dependencies of the speech signal while maintaining a small model size. The proposed Conv-TasNet system significantly outperforms previous time-frequency masking methods in separating two- and three-speaker mixtures. Additionally, Conv-TasNet surpasses several ideal time-frequency magnitude masks in two-speaker speech separation as evaluated by both objective distortion measures and subjective quality assessment by human listeners. Finally, Conv-TasNet has a significantly smaller model size and a much shorter minimum latency, making it a suitable solution for both offline and real-time speech separation applications. This study therefore represents a major step toward the realization of speech separation systems for real-world speech processing technologies.
单通道、与说话者无关的语音分离方法近来取得了巨大进展。然而，此类方法的准确性、延迟和计算成本仍显不足。大多数先前的方法通过混合信号的时频表示来构建分离问题，这存在若干缺点，包括信号的相位和幅度解耦、用于语音分离的时频表示的次优性以及整个系统的长延迟。为解决这些缺点，我们提出了一种全卷积时域音频分离网络（Conv-TasNet），这是一种用于端到端时域语音分离的深度学习框架。Conv-TasNet使用线性编码器来生成针对分离各个说话者而优化的语音波形表示。通过对编码器输出应用一组加权函数（掩码）来实现说话者分离。然后使用线性解码器将修改后的编码器表示转换回波形。使用由堆叠的一维扩张卷积块组成的时域卷积网络（TCN）来找到掩码，这使得网络能够在保持较小模型规模的同时对语音信号的长期依赖性进行建模。所提出的Conv-TasNet系统在分离两说话者和三说话者混合语音方面显著优于先前的时频掩码方法。此外，通过客观失真度量和人类听众的主观质量评估，Conv-TasNet在两说话者语音分离方面超过了几种理想的时频幅度掩码。最后，Conv-TasNet具有显著更小的模型规模和更短的最小延迟，使其成为离线和实时语音分离应用的合适解决方案。因此，本研究朝着实现用于现实世界语音处理技术的语音分离系统迈出了重要一步。
