Takahashi Shuntaro, Tanaka-Ishii Kumiko
The University of Tokyo, Graduate School of Frontier Sciences, Chiba 277-8563, Japan.
The University of Tokyo, Research Center for Advanced Science and Technology, Tokyo 153-8904, Japan.
PLoS One. 2017 Dec 29;12(12):e0189326. doi: 10.1371/journal.pone.0189326. eCollection 2017.
The performance of deep learning in natural language processing has been spectacular, but the reasons for this success remain unclear because of the inherent complexity of deep learning. This paper provides empirical evidence of its effectiveness and of a limitation of neural networks for language engineering. Precisely, we demonstrate that a neural language model based on long short-term memory (LSTM) effectively reproduces Zipf's law and Heaps' law, two representative statistical properties underlying natural language. We discuss the quality of reproducibility and the emergence of Zipf's law and Heaps' law as training progresses. We also point out that the neural language model has a limitation in reproducing long-range correlation, another statistical property of natural language. This understanding could provide a direction for improving the architectures of neural networks.
深度学习在自然语言处理中的表现十分出色，但其成功原因仍不明晰，因为深度学习本身就很复杂。本文提供了关于其有效性以及神经网络在语言工程方面局限性的实证证据。具体而言，我们证明了基于长短期记忆（LSTM）的神经语言模型能有效地重现齐普夫定律和希普斯定律，这是自然语言背后的两个代表性统计特性。我们讨论了重现性的质量以及随着训练的进行齐普夫定律和希普斯定律的出现情况。我们还指出，神经语言模型在重现自然语言的另一个统计特性——长程相关性方面存在局限性。这种认识可为改进神经网络架构提供方向。
