Optimizing Objective Functions Determined from Random Forests

Abstract

We study the problem of optimizing a tree-based ensemble objective with the feasible decisions lie in a polyhedral set. We model this optimization problem as a Mixed Integer Linear Program (MILP). We show this model can be solved to optimality efficiently using Pareto optimal Benders cuts. For large problems, we consider a random forest approximation that consists of only a subset of trees and establish analytically that this gives rise to near optimal solutions by proving analytical guarantees. The error of the approximation decays exponentially as the number of trees increases. Motivated from this result, we propose heuristics that optimize over smaller forests rather than one large one. We present case studies on a property investment problem and a jury selection problem. We show this approach performs well against benchmarks, while providing insights into the sensitivity of the algorithm's performance for different parameters of the random forest.