An adaptive Drop method for deep neural networks regularization: Estimation of DropConnect hyperparameter using generalization gap

Abstract

Deep neural networks (DNNs) have solved numerous challenging reels problems. However, successful DNNs often require a large number of parameters, which may produce some undesirable phenomena, notably overfitting. DropConnect, which is a generalization of Dropout, is one of the successful stochastic regularizers that prevent overfitting in deep neural networks. Indeed, it allows dropping some parameters according to a fixed probability, generating at the end dynamic sparse DNNs. The study of the DropConnect hyperparameter is still not estimated and needs a theoretical understanding. In this context, we propose an estimation of the DropConnect hyperparameter using the gap generalization and the Rademacher complexity. This estimation gives rise to a new DropConnect technique named Adaptive DropConnect (A-DropConnect), in which the studied hyperparameter is updated using the data-dependent during training. Efficiency of A-DropConnect is demonstrated by several experiments. Numerical results demonstrate that the proposed method yields significant improvement in classification performance compared to the state of the art.