IEEE Trans Pattern Anal Mach Intell. 2023 Apr;45(4):4038-4050. doi: 10.1109/TPAMI.2022.3195658. Epub 2023 Mar 7.
Hyperparameter optimization (HPO), characterized by hyperparameter tuning, is not only a critical step for effective modeling but also is the most time-consuming process in machine learning. Traditional search-based algorithms tend to require extensive configuration evaluations for each round to select the desirable hyperparameters during the process, and they are often very inefficient for the implementations on large-scale tasks. In this paper, we study the HPO problem via meta-learning (MtL) approach under the low-rank tensor completion (LRTC) framework. Our proposed approach predicts the performance for hyperparameters of new problems based on their previous performance so that the underlying suitable hyperparameters with better efficiency can be attained. Different from existing approaches, the hyperparameter performance space is instantiated under tensor framework that can preserve the spatial structure and reflect the correlations among the adjacent hyperparameters. When some partial evaluations are available for a new problem, the task of estimating the performance of the unevaluated hyperparameters can be formulated as a tensor completion (TC) problem. Toward the completion purpose, we develop an LRTC algorithm utilizing the sum of nuclear norm (SNN) model. A kernelized version is further developed to capture the nonlinear structure of the performance space. In addition, a corresponding coupled matrix factorization (CMF) algorithm is established to render the predictions solely depend on the meta-features to avoid additional hyperparameter evaluations. Finally, a strategy integrating LRTC and CMF is provided to further enhance the recommendation capacity. We test recommendation performance with our proposed methods for classical SVM and the state-of-the-art deep neural networks such as vision transformer (ViT) and residual network (ResNet), and the obtained results demonstrate the effectiveness of our approaches under various evaluation metrics by comparing with the baselines commonly used for MtL.
超参数优化（HPO）的特点是超参数调整，它不仅是有效建模的关键步骤，也是机器学习中最耗时的过程。传统的基于搜索的算法往往需要对每一轮进行广泛的配置评估，以便在这个过程中选择理想的超参数，因此它们对于大规模任务的实现效率往往非常低。在本文中，我们通过元学习（MtL）方法在低秩张量补全（LRTC）框架下研究 HPO 问题。我们的方法基于它们以前的性能来预测新问题的超参数性能，以便获得更有效率的潜在合适超参数。与现有方法不同，超参数性能空间是在张量框架下实例化的，它可以保留空间结构并反映相邻超参数之间的相关性。当一个新问题有部分评估结果可用时，估计未评估超参数的性能任务可以表示为张量补全（TC）问题。为了实现补全的目的，我们利用核范数和（SNN）模型开发了一个 LRTC 算法。进一步开发了一个核化版本来捕捉性能空间的非线性结构。此外，建立了一个相应的耦合矩阵分解（CMF）算法，以使得预测仅依赖于元特征，从而避免额外的超参数评估。最后，提出了一种集成 LRTC 和 CMF 的策略，以进一步提高推荐能力。我们使用经典 SVM 和最先进的深度神经网络（如视觉转换器（ViT）和残差网络（ResNet））的方法来测试推荐性能，并通过与常用于 MtL 的基线比较，在各种评估指标下验证了我们方法的有效性。
