Graduate School of Informatics, Kyoto University, Engineering Building #10, Sakyo, Kyoto, 606-8501, Japan.
Neural Netw. 2011 May;24(4):311-20. doi: 10.1016/j.neunet.2010.12.006. Epub 2011 Jan 12.
We show that a Multiple Timescale Recurrent Neural Network (MTRNN) can acquire the capabilities to recognize, generate, and correct sentences by self-organizing in a way that mirrors the hierarchical structure of sentences: characters grouped into words, and words into sentences. The model can control which sentence to generate depending on its initial states (generation phase) and the initial states can be calculated from the target sentence (recognition phase). In an experiment, we trained our model over a set of unannotated sentences from an artificial language, represented as sequences of characters. Once trained, the model could recognize and generate grammatical sentences, even if they were not learned. Moreover, we found that our model could correct a few substitution errors in a sentence, and the correction performance was improved by adding the errors to the training sentences in each training iteration with a certain probability. An analysis of the neural activations in our model revealed that the MTRNN had self-organized, reflecting the hierarchical linguistic structure by taking advantage of the differences in timescale among its neurons: in particular, neurons that change the fastest represented "characters", those that change more slowly, "words", and those that change the slowest, "sentences".
我们展示了一种多时间尺度递归神经网络（MTRNN）可以通过自我组织来获得识别、生成和纠正句子的能力，这种自我组织的方式反映了句子的层次结构：字符组成单词，单词组成句子。该模型可以根据其初始状态来控制要生成的句子（生成阶段），并且可以根据目标句子来计算初始状态（识别阶段）。在一项实验中，我们在一组来自人工语言的未注释句子上训练了我们的模型，这些句子表示为字符序列。一旦训练完成，该模型就可以识别和生成语法正确的句子，即使它们没有被学习过。此外，我们发现我们的模型可以纠正句子中的一些替换错误，并且通过以一定的概率将错误添加到每个训练迭代中的训练句子中，可以提高纠错性能。对我们模型中的神经活动的分析表明，MTRNN 已经自我组织，通过利用其神经元之间的时间尺度差异来反映层次化的语言结构：特别是，变化最快的神经元代表“字符”，变化较慢的神经元代表“单词”，而变化最慢的神经元代表“句子”。
