Emergent scale invariance in neural networks

Abstract

We demonstrate, both analytically and numerically, that learning dynamics of neural networks is generically attracted towards a scale-invariant state. The effect can be modeled with quartic interactions between non-trainable variables (e.g. states of neurons) and trainable variables (e.g. weight matrix). Non-trainable variables are rapidly driven towards stochastic equilibrium and trainable variables are slowly driven towards learning equilibrium described by a scale-invariant distribution on a wide range of scales.