Network Architecture Search Research Articles

Qualities, Challenges and Future of Genetic Algorithms

Genetic algorithms, computer programs that simulate natural evolution, are increasingly applied across many disciplines. They have been used to solve various optimisation problems from neural network architecture search to strategic games, and to model phenomena of adaptation and learning. Expertise on the qualities and drawbacks of this technique is largely scattered across the literature or former, motivating an compilation of this knowledge at the light of the most recent developments of the field. In this review, we present genetic algorithms, their qualities, limitations and challenges, as well as some future development perspectives. Genetic algorithms are capable of exploring large and complex spaces of possible solutions, to quickly locate promising elements, and provide an adequate modelling tool to describe evolutionary systems, from games to economies. They however suffer from high computation costs, difficult parameter configuration, and crucial representation of the solutions. Recent developments such as GPU, parallel and quantum computing, conception of powerful parameter control methods, and novel approaches in representation strategies, may be keys to overcome those limitations. This compiling review aims at informing practitioners and newcomers in the field alike in their genetic algorithm research, and at outlining promising avenues for future research. It highlights the potential for interdisciplinary research associating genetic algorithms to pulse original discoveries in social sciences, open ended evolution, artificial life and AI.

Efficient network architecture search via multiobjective particle swarm optimization based on decomposition

The efforts devoted to manually increasing the width and depth of convolutional neural network (CNN) usually require a large amount of time and expertise. It has stimulated a rising demand of neural architecture search (NAS) over these years. However, most popular NAS approaches solely optimize for low prediction error without penalizing high structure complexity. To this end, this paper proposes MOPSO/D-Net, a CNN architecture search method with multiobjective particle swarm optimization based on decomposition (MOPSO/D). The main goal is to reformulate NAS as a multiobjective evolutionary optimization problem, where the optimal architecture is learned by minimizing two conflicting objectives, namely the error rate of classification and number of parameters of the network. Along with the hybrid binary encoding and adaptive penalty-based boundary intersection, an improved MOPSO/D is further proposed to solve the formulated multiobjective NAS and provide diverse tradeoff solutions. Experimental studies verify the effectiveness of MOPSO/D-Net compared with current manual and automated CNN generation methods. The proposed algorithm achieves impressive classification performance with a small number of parameters on each of two benchmark datasets, particularly, 0.4% error rate with 0.16M params on MNIST and 5.88% error rate with 8.1M params on CIFAR-10, respectively.

GenSyth: a new way to understand deep learning

GenSyth: a new way to understand deep learning

TAPAS: Train-Less Accuracy Predictor for Architecture Search

In recent years an increasing number of researchers and practitioners have been suggesting algorithms for large-scale neural network architecture search: genetic algorithms, reinforcement learning, learning curve extrapolation, and accuracy predictors. None of them, however, demonstrated highperformance without training new experiments in the presence of unseen datasets. We propose a new deep neural network accuracy predictor, that estimates in fractions of a second classification performance for unseen input datasets, without training. In contrast to previously proposed approaches, our prediction is not only calibrated on the topological network information, but also on the characterization of the dataset-difficulty which allows us to re-tune the prediction without any training. Our predictor achieves a performance which exceeds 100 networks per second on a single GPU, thus creating the opportunity to perform large-scale architecture search within a few minutes. We present results of two searches performed in 400 seconds on a single GPU. Our best discovered networks reach 93.67% accuracy for CIFAR-10 and 81.01% for CIFAR-100, verified by training. These networks are performance competitive with other automatically discovered state-of-the-art networks however we only needed a small fraction of the time to solution and computational resources.

Native Language Identification in Very Short Utterances Using Bidirectional Long Short-Term Memory Network

Native language identification (NLI) is the task of identifying the first language of a user based on their speech or written text in a second language. In this paper, we propose the use of spectrogram- and cochleagram-based features extracted from very short speech utterances (0.8 s on average) to infer the native language of an Urdu speaker. The bidirectional long short-term memory (BLSTM) neural networks are adopted for the classification of utterances among the native languages. A set of experiments is carried out for the network architecture search and the system’s accuracy is evaluated on the validation data set. Overall accuracy of 74.81% and 71.61% is achieved using the Mel-frequency cepstral coefficients (MFCC) and Gammatone frequency cepstral coefficients (GFCC), respectively. Moreover, the optimized MFCC feature-based BLSTM network and GFCC feature-based BLSTM network are merged together to take advantage of both the feature sets. The experiments show that the performance of the merged network surpasses the individual BLSTM networks and accuracy of 75.69% is achieved on the evaluation data. The effect of test data duration is also analyzed (from 0.27 s to 1.5 s); in addition, it is observed that with very short duration as 0.4 s, an accuracy of over 50% can be achieved.