Inoue Shuichi, Nobukawa Sou, Nishimura Haruhiko, Watanabe Eiji, Isokawa Teijiro
Graduate School of Information and Computer Science, Chiba Institute of Technology, Narashino, Japan.
LY Corporation, Chiyoda-ku, Japan.
Front Artif Intell. 2024 Jul 16;7:1397915. doi: 10.3389/frai.2024.1397915. eCollection 2024.
The deep echo state network (Deep-ESN) architecture, which comprises a multi-layered reservoir layer, exhibits superior performance compared to conventional echo state networks (ESNs) owing to the divergent layer-specific time-scale responses in the Deep-ESN. Although researchers have attempted to use experimental trial-and-error grid searches and Bayesian optimization methods to adjust the hyperparameters, suitable guidelines for setting hyperparameters to adjust the time scale of the dynamics in each layer from the perspective of dynamical characteristics have not been established. In this context, we hypothesized that evaluating the dependence of the multi-time-scale dynamical response on the leaking rate as a typical hyperparameter of the time scale in each neuron would help to achieve a guideline for optimizing the hyperparameters of the Deep-ESN.
First, we set several leaking rates for each layer of the Deep-ESN and performed multi-scale entropy (MSCE) analysis to analyze the impact of the leaking rate on the dynamics in each layer. Second, we performed layer-by-layer cross-correlation analysis between adjacent layers to elucidate the structural mechanisms to enhance the performance.
As a result, an optimum task-specific leaking rate value for producing layer-specific multi-time-scale responses and a queue structure with layer-to-layer signal transmission delays for retaining past applied input enhance the Deep-ESN prediction performance.
These findings can help to establish ideal design guidelines for setting the hyperparameters of Deep-ESNs.
深度回声状态网络（Deep-ESN）架构包含一个多层储备层，由于其各层特定时间尺度响应的差异，与传统回声状态网络（ESN）相比表现出卓越的性能。尽管研究人员已尝试使用实验试错网格搜索和贝叶斯优化方法来调整超参数，但尚未从动力学特性的角度建立合适的准则来设置超参数，以调整各层动力学的时间尺度。在此背景下，我们假设评估多时间尺度动力学响应与作为每个神经元时间尺度典型超参数的泄漏率之间的依赖性，将有助于实现优化Deep-ESN超参数的准则。
首先，我们为Deep-ESN的每一层设置了几个泄漏率，并进行多尺度熵（MSCE）分析，以分析泄漏率对各层动力学的影响。其次，我们对相邻层进行逐层互相关分析，以阐明增强性能的结构机制。
结果表明，用于产生特定层多时间尺度响应的最佳任务特定泄漏率值，以及用于保留过去应用输入的具有层间信号传输延迟的队列结构，可提高Deep-ESN的预测性能。
这些发现有助于建立设置Deep-ESN超参数的理想设计准则。
