TREAT: a trust-region-based error-aggregated training algorithm for neural networks

Abstract

A trust-region-based error-aggregated training algorithm (TREAT) for multi-layer feedforward neural networks is proposed. In the same spirit as that of the Levenberg-Marquardt (LM) method, the TREAT algorithm uses a different Hessian matrix approximation, which is based on the Jacobian matrix derived from aggregated errors. An aggregation scheme is discussed. It can greatly reduce the size of the matrix to be inverted in each training iteration and thereby lower the iterative computational cost. Compared with the LM method, the TREAT algorithm is computationally less intensive, and requires less memory. This is especially important for large sized neural networks where the LM algorithm becomes impractical.