Complete and representative training of neural networks: A generalization study using double noise injection and natural images

Abstract

Neural networks hold substantial promise for automating various processing and interpretation tasks. Yet, their performance is often suboptimal compared with standard but more closely guided approaches. The lack of performance is often attributed to poor generalization, in particular if fewer training examples are provided than free parameters that exist in the machine-learning algorithm. In this case, the training data are typically memorized instead of the algorithm learning the underlying general trends. The network generalization is improved if the provided samples are representative, in that they describe all features of interest well. We argue that a more subtle condition preventing poor performance is that the provided examples must also be complete; the examples must span the full solution space. Ensuring completeness during training is challenging unless the target application is well understood. We illustrate that one possible solution is to make the problem more general if this greatly increases the number of available training data. For instance, if seismic images are treated as a subclass of natural images, then a deep-learning-based denoiser for seismic data can be trained using exclusively natural images. The latter are widely available. The resulting denoising algorithm has never seen any seismic data during the training stage, yet it displays performance comparable with standard and advanced random-noise reduction methods. We exclude any seismic data during training to demonstrate that natural images are complete and representative for this specific task. Furthermore, we apply a novel approach to increase the amount of training data known as double noise injection, providing noisy input and output images during the training process. Given the importance of network generalization, we hope that the insights that we gained may help improve the performance of a range of machine-learning applications in geophysics.