Hewamalage Hansika, Ackermann Klaus, Bergmeir Christoph
School of Computer Science & Engineering, University of New South Wales, Sydney, Australia.
SoDa Labs and Department of Econometrics & Business Statistics, Monash Business School, Monash University, Melbourne, Australia.
Data Min Knowl Discov. 2023;37(2):788-832. doi: 10.1007/s10618-022-00894-5. Epub 2022 Dec 2.
Recent trends in the Machine Learning (ML) and in particular Deep Learning (DL) domains have demonstrated that with the availability of massive amounts of time series, ML and DL techniques are competitive in time series forecasting. Nevertheless, the different forms of non-stationarities associated with time series challenge the capabilities of data-driven ML models. Furthermore, due to the domain of forecasting being fostered mainly by statisticians and econometricians over the years, the concepts related to forecast evaluation are not the mainstream knowledge among ML researchers. We demonstrate in our work that as a consequence, ML researchers oftentimes adopt flawed evaluation practices which results in spurious conclusions suggesting methods that are not competitive in reality to be seemingly competitive. Therefore, in this work we provide a tutorial-like compilation of the details associated with forecast evaluation. This way, we intend to impart the information associated with forecast evaluation to fit the context of ML, as means of bridging the knowledge gap between traditional methods of forecasting and adopting current state-of-the-art ML techniques.We elaborate the details of the different problematic characteristics of time series such as non-normality and non-stationarities and how they are associated with common pitfalls in forecast evaluation. Best practices in forecast evaluation are outlined with respect to the different steps such as data partitioning, error calculation, statistical testing, and others. Further guidelines are also provided along selecting valid and suitable error measures depending on the specific characteristics of the dataset at hand.
机器学习（ML）领域，尤其是深度学习（DL）领域的最新趋势表明，随着大量时间序列数据的可得性，ML和DL技术在时间序列预测方面具有竞争力。然而，与时间序列相关的不同形式的非平稳性对数据驱动的ML模型的能力提出了挑战。此外，由于多年来预测领域主要由统计学家和计量经济学家主导，与预测评估相关的概念并非ML研究人员的主流知识。我们在工作中证明，因此，ML研究人员常常采用有缺陷的评估方法，这导致得出虚假结论，即一些在实际中并无竞争力的方法看似具有竞争力。所以，在这项工作中，我们提供了一份类似教程的内容，汇编了与预测评估相关的细节。通过这种方式，我们旨在传授与预测评估相关的信息，使其符合ML的背景，以此弥合传统预测方法与采用当前最先进ML技术之间的知识差距。我们详细阐述了时间序列的不同问题特征，如非正态性和非平稳性，以及它们如何与预测评估中的常见陷阱相关联。针对数据划分、误差计算、统计检验等不同步骤概述了预测评估的最佳实践。还根据手头数据集的特定特征，提供了选择有效且合适的误差度量的进一步指导方针。
