Peter the Great St. Petersburg Polytechnic University, St. Petersburg, Russia.
Neural Netw. 2022 Oct;154:346-359. doi: 10.1016/j.neunet.2022.07.029. Epub 2022 Aug 1.
A new approach called ABRF (the attention-based random forest) and its modifications for applying the attention mechanism to the random forest (RF) for regression and classification are proposed. The main idea behind the proposed ABRF models is to assign attention weights with trainable parameters to decision trees in a specific way. The attention weights depend on the distance between an instance, which falls into a corresponding leaf of a tree, and training instances, which fall in the same leaf. This idea stems from representation of the Nadaraya-Watson kernel regression in the form of a RF. Three modifications of the general approach are proposed. The first one is based on applying the Huber's contamination model and on computing the attention weights by solving quadratic or linear optimization problems. The second and the third modifications use the gradient-based algorithms for computing an extended set of the attention trainable parameters. Numerical experiments with various regression and classification datasets illustrate the proposed method. The code implementing the approach is publicly available.
提出了一种新的方法，称为基于注意力的随机森林（ABRF）及其对随机森林（RF）的修改，以便将注意力机制应用于回归和分类。所提出的 ABRF 模型的主要思想是以特定的方式为决策树分配具有可训练参数的注意力权重。注意力权重取决于实例之间的距离，该实例落入树的相应叶子中，而训练实例则落入同一叶子中。这个想法源于以 RF 形式表示的 Nadaraya-Watson 核回归。提出了三种通用方法的修改。第一种方法基于应用 Huber 的污染模型，并通过求解二次或线性优化问题来计算注意力权重。第二和第三种修改使用基于梯度的算法来计算可训练注意力参数的扩展集。使用各种回归和分类数据集的数值实验说明了所提出的方法。实现该方法的代码是公开的。
