Cognitive Systems Lab, University of Bremen, Bremen, Germany.
Department of Neurosurgery, School of Mental Health and Neurosciences, Maastricht University, Maastricht, The Netherlands.
Commun Biol. 2021 Sep 23;4(1):1055. doi: 10.1038/s42003-021-02578-0.
Speech neuroprosthetics aim to provide a natural communication channel to individuals who are unable to speak due to physical or neurological impairments. Real-time synthesis of acoustic speech directly from measured neural activity could enable natural conversations and notably improve quality of life, particularly for individuals who have severely limited means of communication. Recent advances in decoding approaches have led to high quality reconstructions of acoustic speech from invasively measured neural activity. However, most prior research utilizes data collected during open-loop experiments of articulated speech, which might not directly translate to imagined speech processes. Here, we present an approach that synthesizes audible speech in real-time for both imagined and whispered speech conditions. Using a participant implanted with stereotactic depth electrodes, we were able to reliably generate audible speech in real-time. The decoding models rely predominately on frontal activity suggesting that speech processes have similar representations when vocalized, whispered, or imagined. While reconstructed audio is not yet intelligible, our real-time synthesis approach represents an essential step towards investigating how patients will learn to operate a closed-loop speech neuroprosthesis based on imagined speech.
言语神经修复旨在为因身体或神经损伤而无法说话的个体提供一种自然的交流渠道。直接从测量到的神经活动实时合成声学语音，可以实现自然对话，并显著提高生活质量，特别是对于那些交流手段严重受限的个体。解码方法的最新进展已经能够从侵入性测量的神经活动中重建出高质量的声学语音。然而，大多数先前的研究都利用了在有意义的语音实验中收集的数据，这些数据可能不能直接转化为想象中的语音过程。在这里，我们提出了一种可以实时合成想象和低语语音的方法。使用植入立体定向深度电极的参与者，我们能够实时可靠地生成可听见的语音。解码模型主要依赖于额部活动，这表明当发声、低语或想象时，言语过程具有相似的表示。虽然重建的音频还不能理解，但我们的实时合成方法代表了朝着研究患者如何基于想象中的语音来学习操作闭环言语神经修复的重要一步。
