Exploring Quaternion Neural Network Loss Surfaces

Abstract

This paper explores the superior performance of quaternion multi-layer perceptron (QMLP) neural networks over real-valued multi-layer perceptron (MLP) neural networks, a phenomenon that has been empirically observed but not thoroughly investigated. The study utilizes loss surface visualization and projection techniques to examine quaternion-based optimization loss surfaces for the first time. The primary contribution of this research is the statistical evidence that QMLP models yield smoother loss surfaces than real-valued neural networks, which are measured and compared using a robust quantitative measure of loss surface “goodness” based on estimates of surface curvature. Extensive computational testing validates the effectiveness of these surface curvature estimates. The paper presents a comprehensive comparison of the average surface curvature of a tuned QMLP model and a tuned real-valued MLP model on both a regression task and a classification task. The results provide strong support for the improved optimization performance observed in QMLPs across various problem domains.