Particle swarm optimization for hyper-parameter selection in deep neural networks

Abstract

Deep neural networks (DNNs) have achieved unprecedented success in a wide array of tasks. However, the performance of these systems depends directly on their hyper-parameters which often must be selected by an expert. Optimizing the hyper-parameters remains a substantial obstacle in designing DNNs in practice. In this work, we propose to select them using particle swarm optimization (PSO). Such biologically-inspired approaches have not been extensively exploited for this task. We demonstrate that PSO efficiently explores the solution space, allowing DNNs of a minimal topology to obtain competitive classification performance over the MNIST dataset. We showed that very small DNNs optimized by PSO retrieve promising classification accuracy for CIFAR-10. Also, PSO improves the performance of existing architectures. Extensive experimental study, backed-up with the statistical tests, revealed that PSO is an effective technique for automating hyper-parameter selection and efficiently exploits computational resources.