Abstract
We investigate the effect of explicitly enforcing the Lipschitz continuity of neural networks with respect to their inputs. To this end, we provide a simple technique for computing an upper bound to the Lipschitz constant—for multiple p-norms—of a feed forward neural network composed of commonly used layer types. Our technique is then used to formulate training a neural network with a bounded Lipschitz constant as a constrained optimisation problem that can be solved using projected stochastic gradient methods. Our evaluation study shows that the performance of the resulting models exceeds that of models trained with other common regularisers. We also provide evidence that the hyperparameters are intuitive to tune, demonstrate how the choice of norm for computing the Lipschitz constant impacts the resulting model, and show that the performance gains provided by our method are particularly noticeable when only a small amount of training data is available.
Highlights
Supervised learning is primarily concerned with the problem of approximating a function given examples of what output should be produced for a particular input
The results on CIFAR-100 follow a similar trend to those observed on CIFAR-10: Lipschitz constant constraint (LCC) performs the best, dropout provides a small increase in performance over no regularisation, and combining dropout other approaches can sometimes provide a small boost in accuracy
This paper has presented a simple and effective regularisation technique for deep feed-forward neural networks called Lipschitz constant constraint (LCC), shown that it is applicable to a variety of feed-forward neural network architectures, and established that it is suited to situations where only a small amount of training data is available
Summary
Supervised learning is primarily concerned with the problem of approximating a function given examples of what output should be produced for a particular input. Machine Learning (2021) 110:393–416 we need to select an appropriate space of functions in which the machine should search for a good approximation, and select an algorithm to search through this space This is typically done by first picking a large family of models, such as support vector machines or decision trees, and applying a suitable search algorithm. Well-understood regularisation approaches adapted from linear models, such as applying an 2 penalty term to the model parameters, are known to be less effective than the heuristic approaches (Srivastava et al 2014). This provides a clear motivation for developing well-founded and effective regularisation methods for neural networks.
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Free) 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call
