Tang Lingyu, Wang Jun, Wang Mengyao, Zhao Chunyu
School of Science, Civil Aviation Flight University of China, Guanghan 618307, China.
Business School, Sichuan Normal University, Chengdu 610101, China.
Entropy (Basel). 2024 Feb 28;26(3):215. doi: 10.3390/e26030215.
The echo state network (ESN) is a recurrent neural network that has yielded state-of-the-art results in many areas owing to its rapid learning ability and the fact that the weights of input neurons and hidden neurons are fixed throughout the learning process. However, the setting procedure for initializing the ESN's recurrent structure may lead to difficulties in designing a sound reservoir that matches a specific task. This paper proposes an improved pre-training method to adjust the model's parameters and topology to obtain an adaptive reservoir for a given application. Two strategies, namely global random selection and ensemble training, are introduced to pre-train the randomly initialized ESN model. Specifically, particle swarm optimization is applied to optimize chosen fixed and global weight values within the network, and the reliability and stability of the pre-trained model are enhanced by employing the ensemble training strategy. In addition, we test the feasibility of the model for time series prediction on six benchmarks and two real-life datasets. The experimental results show a clear enhancement in the ESN learning results. Furthermore, the proposed global random selection and ensemble training strategies are also applied to pre-train the extreme learning machine (ELM), which has a similar training process to the ESN model. Numerical experiments are subsequently carried out on the above-mentioned eight datasets. The experimental findings consistently show that the performance of the proposed pre-trained ELM model is also improved significantly. The suggested two strategies can thus enhance the ESN and ELM models' prediction accuracy and adaptability.
回声状态网络（ESN）是一种递归神经网络，由于其快速学习能力以及输入神经元和隐藏神经元的权重在整个学习过程中保持固定这一事实，它在许多领域都取得了领先的成果。然而，初始化ESN递归结构的设置过程可能会导致在设计与特定任务匹配的合理储备池时遇到困难。本文提出了一种改进的预训练方法，用于调整模型的参数和拓扑结构，以获得适用于给定应用的自适应储备池。引入了两种策略，即全局随机选择和集成训练，对随机初始化的ESN模型进行预训练。具体而言，应用粒子群优化算法来优化网络内选定的固定权重值和全局权重值，并通过采用集成训练策略提高预训练模型的可靠性和稳定性。此外，我们在六个基准数据集和两个实际生活数据集上测试了该模型进行时间序列预测的可行性。实验结果表明ESN的学习结果有了明显提升。此外，所提出的全局随机选择和集成训练策略也被应用于预训练极限学习机（ELM），其训练过程与ESN模型类似。随后在上述八个数据集上进行了数值实验。实验结果一致表明，所提出的预训练ELM模型的性能也得到了显著提高。因此，所建议的两种策略可以提高ESN和ELM模型的预测准确性和适应性。
