Annu Int Conf IEEE Eng Med Biol Soc. 2022 Jul;2022:3100-3104. doi: 10.1109/EMBC48229.2022.9871301.
Speech decoding from brain activity can enable development of brain-computer interfaces (BCIs) to restore naturalistic communication in paralyzed patients. Previous work has focused on development of decoding models from isolated speech data with a clean background and multiple repetitions of the material. In this study, we describe a novel approach to speech decoding that relies on a generative adversarial neural network (GAN) to reconstruct speech from brain data recorded during a naturalistic speech listening task (watching a movie). We compared the GAN-based approach, where reconstruction was done from the compressed latent representation of sound decoded from the brain, with several baseline models that reconstructed sound spectrogram directly. We show that the novel approach provides more accurate reconstructions compared to the baselines. These results underscore the potential of GAN models for speech decoding in naturalistic noisy environments and further advancing of BCIs for naturalistic communication. Clinical Relevance - This study presents a novel speech decoding paradigm that combines advances in deep learning, speech synthesis and neural engineering, and has the potential to advance the field of BCI for severely paralyzed individuals.
从大脑活动中进行语音解码可以开发脑机接口（BCI），以恢复瘫痪患者的自然语言交流。以前的研究主要集中在开发从背景干净且材料有多次重复的孤立语音数据中解码模型。在这项研究中，我们描述了一种新颖的语音解码方法，该方法依赖于生成对抗神经网络（GAN），从自然语言聆听任务（观看电影）期间记录的大脑数据中重建语音。我们比较了基于 GAN 的方法，其中从大脑解码的声音的压缩潜在表示中进行重建，以及直接重建声音频谱图的几种基线模型。我们表明，与基线相比，新方法提供了更准确的重建。这些结果强调了 GAN 模型在自然噪声环境中的语音解码中的潜力，并进一步推进了用于自然语言交流的 BCI。临床相关性-本研究提出了一种新颖的语音解码范式，结合了深度学习、语音合成和神经工程的进步，有潜力推进严重瘫痪患者的 BCI 领域。
