Feng Chen, Cao Lu, Wu Di, Zhang En, Wang Ting, Jiang Xiaowei, Chen Jinbo, Wu Hui, Lin Siyu, Hou Qiming, Zhu Junming, Yang Jie, Sawan Mohamad, Zhang Yue
Department of Neurosurgery, The Second Affiliated Hospital of Zhejiang University School of Medicine, Hangzhou, Zhejiang Province, China.
School of Engineering, Westlake University, Hangzhou, Zhejiang Province, China.
Cyborg Bionic Syst. 2025 Apr 29;6:0257. doi: 10.34133/cbsystems.0257. eCollection 2025.
Recent advances in brain-computer interfaces (BCIs) have demonstrated the potential to decode language from brain activity into sound or text, which has predominantly focused on alphabetic languages, such as English. However, logosyllabic languages, such as Mandarin Chinese, present marked challenges for establishing decoders that cover all characters, due to its unique syllable structures, extended character sets (e.g., over 50,000 characters for Mandarin Chinese), and complex mappings between characters and syllables, thus hindering practical applications. Here, we leverage the acoustic features of Mandarin Chinese syllables, constructing prediction models for syllable components (initials, tones, and finals), and decode speech-related stereoelectroencephalography (sEEG) signals into coherent Chinese sentences. The results demonstrate a high sentence-level offline decoding performance with a median character accuracy of 71.00% over the full spectrum of characters in the best participant. We also verified that incorporating acoustic-related features into the design of prediction models substantially enhances the accuracy of initials, tones, and finals. Moreover, our findings revealed that effective speech decoding also involves subcortical structures like the thalamus in addition to traditional language-related brain regions. Overall, we established a brain-to-sentence decoder for logosyllabic languages over full character set with a large intracranial electroencephalography dataset.
脑机接口（BCI）的最新进展已证明从大脑活动解码语言为声音或文本的潜力，这主要集中在字母语言上，如英语。然而，像汉语这样的语素音节文字语言，由于其独特的音节结构、庞大的字符集（例如，汉语有超过50,000个字符）以及字符与音节之间复杂的映射关系，在建立涵盖所有字符的解码器方面面临显著挑战，从而阻碍了实际应用。在此，我们利用汉语普通话音节的声学特征，构建音节成分（声母、声调、韵母）的预测模型，并将与语音相关的立体脑电图（sEEG）信号解码为连贯的中文句子。结果表明，在最佳参与者的所有字符全谱上，句子级离线解码性能较高，中位数字符准确率为71.00%。我们还验证了将与声学相关的特征纳入预测模型设计中可显著提高声母、声调及韵母的准确率。此外，我们的研究结果表明，有效的语音解码除了涉及传统的与语言相关的脑区外，还涉及丘脑等皮层下结构。总体而言，我们利用一个大型颅内脑电图数据集建立了一种针对语素音节文字语言全字符集的脑到句子解码器。
