Random Forests with Attentive Nodes

Abstract

An extension of the attention-based random forest called ABRF-E for incorporating the attention mechanism into the random forest (RF) for improving the machine learning model performance is proposed. The main idea behind the proposed ABRF-E model is based on the Nadaraya-Watson kernel regression model. The attention weights with trainable parameters are assigned to each node of decision trees. The weights depend on the distance between an instance, which falls into a corresponding node of a tree, and instances, which fall in the same node. The attention weight are computed by using the Huber's contamination model which provides the linear relationship between the attention weights and their trainable parameters. The linear relationship leads to solving a quadratic optimization problem and allows us to avoid gradient-based algorithms for training attention parameters and attention weights. Numerical experiments with real datasets illustrate the proposed model.