Department of Engineering, University of Cambridge, Trumpington Street, Cambridge CB2 1PZ, UK.
Department of Engineering, University of Cambridge, Trumpington Street, Cambridge CB2 1PZ, UK; Aeronautics Department, Imperial College London, Exhibition Rd, London, SW7 2AZ, UK; The Alan Turing Institute, 96 Euston Road, London, England, NW1 2DB, UK; Institute for Advanced Study, Technical University of Munich, Lichtenbergstrasse 2a, 85748 Garching, Germany(1).
Neural Netw. 2021 Oct;142:252-268. doi: 10.1016/j.neunet.2021.05.004. Epub 2021 May 14.
An approach to the time-accurate prediction of chaotic solutions is by learning temporal patterns from data. Echo State Networks (ESNs), which are a class of Reservoir Computing, can accurately predict the chaotic dynamics well beyond the predictability time. Existing studies, however, also showed that small changes in the hyperparameters may markedly affect the network's performance. The overarching aim of this paper is to improve the robustness in the selection of hyperparameters in Echo State Networks for the time-accurate prediction of chaotic solutions. We define the robustness of a validation strategy as its ability to select hyperparameters that perform consistently between validation and test sets. The goal is three-fold. First, we investigate routinely used validation strategies. Second, we propose the Recycle Validation, and the chaotic versions of existing validation strategies, to specifically tackle the forecasting of chaotic systems. Third, we compare Bayesian optimization with the traditional grid search for optimal hyperparameter selection. Numerical tests are performed on prototypical nonlinear systems that have chaotic and quasiperiodic solutions, such as the Lorenz and Lorenz-96 systems, and the Kuznetsov oscillator. Both model-free and model-informed Echo State Networks are analysed. By comparing the networks' performance in learning chaotic (unpredictable) versus quasiperiodic (predictable) solutions, we highlight fundamental challenges in learning chaotic solutions. The proposed validation strategies, which are based on the dynamical systems properties of chaotic time series, are shown to outperform the state-of-the-art validation strategies. Because the strategies are principled - they are based on chaos theory such as the Lyapunov time - they can be applied to other Recurrent Neural Networks architectures with little modification. This work opens up new possibilities for the robust design and application of Echo State Networks, and Recurrent Neural Networks, to the time-accurate prediction of chaotic systems.
一种用于准确预测混沌解的方法是通过从数据中学习时间模式。回声状态网络 (ESN) 是一种 Reservoir Computing 类，可以在可预测时间之外准确地预测混沌动力学。然而，现有研究也表明，超参数的微小变化可能会显著影响网络的性能。本文的总体目标是提高回声状态网络中超参数选择的鲁棒性，以实现对混沌解的准确预测。我们将验证策略的鲁棒性定义为其在验证集和测试集之间选择性能一致的超参数的能力。目标有三个。首先，我们研究了常用的验证策略。其次，我们提出了循环验证策略和现有验证策略的混沌版本，以专门解决混沌系统的预测问题。第三，我们比较了贝叶斯优化与传统的网格搜索，以进行最优超参数选择。在具有混沌和拟周期解的原型非线性系统上进行了数值测试，例如 Lorenz 和 Lorenz-96 系统以及 Kuznetsov 振荡器。分析了无模型和基于模型的回声状态网络。通过比较网络在学习混沌（不可预测）和拟周期（可预测）解方面的性能，我们强调了学习混沌解的基本挑战。所提出的验证策略基于混沌时间序列的动力系统特性，优于最新的验证策略。由于这些策略是基于原则的 - 它们基于混沌理论，例如 Lyapunov 时间 - 因此可以在稍加修改的情况下应用于其他递归神经网络架构。这项工作为回声状态网络和递归神经网络在混沌系统的准确预测中的稳健设计和应用开辟了新的可能性。
