Interpreting the Basis Path Set in Neural Networks

Abstract

The \({\cal G}\)-SGD algorithm significantly outperforms the conventional SGD algorithm in ReLU neural networks by adopting the basis path set. However, how the inner mechanism of basis paths works remains mysterious, and the \({\cal G}\)-SGD algorithm that helps to find a basis path set is heuristic. This paper employs graph theory to investigate structure properties of basis paths in a more general and complicated neural network with unbalanced layers and edge-skipping. The hierarchical Algorithm HBPS is proposed to find a basis path set, by decomposing the complicated network into several independent and parallel substructures. The paper theoretically extends the study of basis paths and provides one methodology to find the basis path set in a more general neural network.