Tan Jia Mian, Liao Haoran, Liu Wei, Fan Changjun, Huang Jincai, Liu Zhong, Yan Junchi
Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, China.
College of Systems Engineering, National University of Defense Technology, Changsha, China.
Math Biosci Eng. 2024 Jun 14;21(6):6289-6335. doi: 10.3934/mbe.2024275.
Hyperparameter optimization (HPO) has been well-developed and evolved into a well-established research topic over the decades. With the success and wide application of deep learning, HPO has garnered increased attention, particularly within the realm of machine learning model training and inference. The primary objective is to mitigate the challenges associated with manual hyperparameter tuning, which can be ad-hoc, reliant on human expertise, and consequently hinders reproducibility while inflating deployment costs. Recognizing the growing significance of HPO, this paper surveyed classical HPO methods, approaches for accelerating the optimization process, HPO in an online setting (dynamic algorithm configuration, DAC), and when there is more than one objective to optimize (multi-objective HPO). Acceleration strategies were categorized into multi-fidelity, bandit-based, and early stopping; DAC algorithms encompassed gradient-based, population-based, and reinforcement learning-based methods; multi-objective HPO can be approached via scalarization, metaheuristics, and model-based algorithms tailored for multi-objective situation. A tabulated overview of popular frameworks and tools for HPO was provided, catering to the interests of practitioners.
超参数优化（HPO）在过去几十年中得到了充分发展，并演变成一个成熟的研究课题。随着深度学习的成功和广泛应用，HPO受到了越来越多的关注，特别是在机器学习模型训练和推理领域。其主要目标是缓解与手动超参数调整相关的挑战，手动调整可能是临时的，依赖于人类专业知识，因此会阻碍可重复性，同时增加部署成本。认识到HPO日益增长的重要性，本文对经典的HPO方法、加速优化过程的方法、在线设置中的HPO（动态算法配置，DAC）以及存在多个优化目标时的情况（多目标HPO）进行了综述。加速策略分为多保真度、基于策略和提前停止；DAC算法包括基于梯度、基于群体和基于强化学习的方法；多目标HPO可以通过标量化、元启发式算法以及针对多目标情况量身定制的基于模型的算法来实现。本文还提供了一个表格化的流行HPO框架和工具概述，以满足从业者的兴趣。
