Dynamical Systems Approach to Outlier Robust Deep Neural Networks for Regression

Abstract

We study the dynamics and equilibria induced by training an artificial neural network for regression based on the gradient conjugate prior (GCP) updates. We show that contaminating the training data set by outliers leads to bifurcation of a stable equilibrium from infinity. Furthermore, using the outputs of the GCP network at the equilibrium, we derive an explicit formula for correcting the learned variance of the marginal distribution and removing the bias caused by outliers in the training set. Assuming a Gaussian (input-dependent) ground truth distribution contaminated with a proportion $\varepsilon$ of outliers, we show that the fitted mean is in a $c e^{-1/\varepsilon}$-neighborhood of the ground truth mean and the corrected variance is in a $b\varepsilon$-neighborhood of the ground truth variance, whereas the uncorrected variance of the marginal distribution can even be infinite. We explicitly find $b$ as a function of the output of the GCP network, without a priori knowledge of the outliers (possibly input-dependent) distribution. Experiments with synthetic and real-world data sets indicate that the GCP network fitted with a standard optimizer outperforms other robust methods for regression.