Original Search Space Research Articles

Particle Swarm Optimization for Efficiently Evolving Deep Convolutional Neural Networks Using an Autoencoder-Based Encoding Strategy

Deep convolutional neural networks (DCNNs) have achieved surpassing success in the field of computer vision, and a number of elaborately designed networks refresh the performance records in benchmark datasets. Recently, evolutionary neural architecture search (ENAS) has become an emerging area, which could employ an evolutionary computation (EC) technique to automatically construct promising network architectures without human intervention. However, existing algorithms still have limitations: most standard EC approaches cannot process flexible-length architecture representations directly, and most fitness evaluation processes suffer from the exhibitive computational cost and the unreliable prediction results. To overcome these shortcomings, we propose an efficient particle swarm optimization (PSO)-based neural architecture search algorithm to search for appropriate dense blocks on image classification tasks, named EAEPSO. EAEPSO addresses the first limitation by designing an autoencoder to encode variable-length network representations as fixed-length latent vectors, which converts the original search space to a latent space that can facilitate the downstream search. Besides, an efficient and effective hierarchical fitness evaluation method is designed to guide the search process to address the second limitation. The experimental results show that EAEPSO is a very competitive ENAS algorithm that achieves an error rate of 2.74% on CIFAR-10 and 16.17% on CIFAR-100, and reduces the computational cost from hundreds or thousands of GPU-days to only 2.2 and 4 GPU-days, respectively. Further analyses investigate the reduced training data’s effect and confirm the effectiveness of both the proposed autoencoder and the proposed hierarchical fitness evaluation method.

On the use of tandem mass spectra acquired from samples of evolutionarily distant organisms to validate methods for false discovery rate estimation.

Estimating the false discovery rate (FDR) of peptide identifications is a key step in proteomics data analysis, and many methods have been proposed for this purpose. Recently, an entrapment-inspired protocol to validate methods for FDR estimation appeared in articles showcasing new spectral library search tools. That validation approach involves generating incorrect spectral matches by searching spectra from evolutionarily distant organisms (entrapment queries) against the original target search space. Although this approach may appear similar to the solutions using entrapment databases, it represents a distinct conceptual framework whose correctness has not been verified yet. In this viewpoint, we first discussed the background of the entrapment-based validation protocols and then conducted a few simple computational experiments to verify the assumptions behind them. The results reveal that entrapment databases may, in some implementations, be a reasonable choice for validation, while the assumptions underpinning validation protocols based on entrapment queries are likely to be violated in practice. This article also highlights the need for well-designed frameworks for validating FDR estimation methods in proteomics.

An efficient evolutionary architecture search for variational autoencoder with alternating optimization and adaptive crossover

Variational autoencoder is a commonly unsupervised learning model. However, its complex structure hinders the utilization of the network architecture search algorithm to release researchers from tedious manual design. To design excellent architectures automatically, this paper proposes an efficient evolutionary architecture search for variational autoencoder with alternating optimization and adaptive crossover(AOC-VAE). Firstly, to alleviate the problem of large search space when automatically designing variational autoencoders, AOC-VAE designs an alternating optimized search mechanism based on the specific coupling of encoder and decoder in variational autoencoders, which reduces the original huge search space almost to half. Then, AOC-VAE can find quickly the optimal individual in the solution space by designing an adaptive crossover mechanism. In early evolutionary period, the structural differences between individuals are relatively significant, making crossover operations more inclined to exchange structural information between individuals. As evolution progresses, the individual structures in the population tend to be similar, and the exchange of information concentrates on the parameter. Finally, in the optimization process, a fitness evaluation mechanism based on dynamic weights is designed to accurately find out the outstanding individuals under the current optimization goal. Individual fitness in the population is more inclined to be affected by the current optimization goal, thus guiding the population to evolve according to the optimization goal at different stages. AOC-VAE is verified on MNIST, SVHN, CIFAR-10, and CIFAR-100 benchmark datasets and compared with 14 algorithms. The experimental results show that the VAE network structure designed by the AOC-VAE performs well in the image classification task.

Optimisation via encodings: a renormalisation group perspective

Difficult, in particular NP-complete, optimization problems are traditionally solved approximately using search heuristics. These are usually slowed down by the rugged landscapes encountered, because local minima arrest the search process. Cover-encoding maps were devised to circumvent this problem by transforming the original landscape to one that is free of local minima and enriched in near-optimal solutions. By definition, these involve the mapping of the original (larger) search space into smaller subspaces, by processes that typically amount to a form of coarse-graining. In this paper, we explore the details of this coarse-graining using formal arguments, as well as concrete examples of cover-encoding maps, that are investigated analytically as well as computationally. Our results strongly suggest that the coarse-graining involved in cover-encoding maps bears a strong resemblance to that encountered in renormalisation group schemes. Given the apparently disparate nature of these two formalisms, these strong similarities are rather startling, and suggest deep mathematical underpinnings that await further exploration.

PWSNAS: Powering Weight Sharing NAS With General Search Space Shrinking Framework.

Neural architecture search (NAS) depends heavily on an efficient and accurate performance estimator. To speed up the evaluation process, recent advances, like differentiable architecture search (DARTS) and One-Shot approaches, instead of training every model from scratch, train a weight-sharing super-network to reuse parameters among different candidates, in which all child models can be efficiently evaluated. Though these methods significantly boost search efficiency, they inherently suffer from inaccurate and unstable performance estimation. To this end, we propose a general and effective framework for powering weight-sharing NAS, namely, PWSNAS, by shrinking search space automatically, i.e., candidate operators will be discarded if they are less important. With the strategy, our approach can provide a promising search space of a smaller size by progressively simplifying the original search space, which can reduce difficulties for existing NAS methods to find superior architectures. In particular, we present two strategies to guide the shrinking process: detect redundant operators with a new angle-based metric and decrease the degree of weight sharing of a super-network by increasing parameters, which differentiates PWSNAS from existing shrinking methods. Comprehensive analysis experiments on NASBench-201 verify the superiority of our proposed metric over existing accuracy-based and magnitude-based metrics. PWSNAS can easily apply to the state-of-the-art NAS methods, e.g., single path one-shot neural architecture search (SPOS), FairNAS, ProxylessNAS, DARTS, and progressive DARTS (PDARTS). We evaluate PWSNAS and demonstrate consistent performance gains over baseline methods.

Multi-Objective Optimization Method for Signalized Intersections in Intelligent Traffic Network.

Urban intersections are one of the most common sources of traffic congestion. Especially for multiple intersections, an appropriate control method should be able to regulate the traffic flow within the control area. The intersection signal-timing problem is crucial for ensuring efficient traffic operations, with the key issues being the determination of a traffic model and the design of an optimization algorithm. So, an optimization method for signalized intersections integrating a multi-objective model and an NSGAIII-DAE algorithm is established in this paper. Firstly, the multi-objective model is constructed including the usual signal control delay and traffic capacity indices. In addition, the conflict delay caused by right-turning vehicles crossing straight-going non-motor vehicles is considered and combined with the proposed algorithm, enabling the traffic model to better balance the traffic efficiency of intersections without adding infrastructure. Secondly, to address the challenges of diversity and convergence faced by the classic NSGA-III algorithm in solving traffic models with high-dimensional search spaces, a denoising autoencoder (DAE) is adopted to learn the compact representation of the original high-dimensional search space. Some genetic operations are performed in the compressed space and then mapped back to the original search space through the DAE. As a result, an appropriate balance between the local and global searching in an iteration can be achieved. To validate the proposed method, numerical experiments were conducted using actual traffic data from intersections in Jinzhou, China. The numerical results show that the signal control delay and conflict delay are significantly reduced compared with the existing algorithm, and the optimal reduction is 33.7% and 31.3%, respectively. The capacity value obtained by the proposed method in this paper is lower than that of the compared algorithm, but it is also 11.5% higher than that of the current scheme in this case. The comparisons and discussions demonstrate the effectiveness of the proposed method designed for improving the efficiency of signalized intersections.

Adaptive lightweight convolutional neural architecture search for segmentation problem

Convolutional neural networks (CNNs) have been shown to have the most advanced performance on image segmentation tasks. However, designing a reasonable convolutional neural network architecture requires a great deal of empirical knowledge, which can be expensive or unavailable in a variety of actual applications. To handle this problem, neural architecture search (NAS) has been developed by simultaneously optimizing the network architecture and hyperparameters, the main disadvantage of which is the requirement for extremely high computational resources. To accelerate the NAS of CNNs, a two-stage adaptive lightweight convolutional neural architecture search (AL-CNAS) method is proposed in this article based on an evolutionary algorithm. In this method, the original search space is divided into two parts based on feature grouping, and only one part is optimized at each stage to achieve an exponential decrease in the number of genotype combinations, which helps to improve search efficiency and reduce computational resource requirements. The proposed AL-CNAS is validated on the real-world segmentation of steel microstructure and two benchmark segmentation problems, and the computational results show that the proposed AL-CNAS has good segmentation performance and generality.

Orthogonal Transfer for Multitask Optimization

Knowledge transfer (KT) plays a key role in multitask optimization. However, most of the existing KT methods still face two challenges. First, the tasks may commonly have different dimensionalities (DDs), making the KT between heterogeneous search spaces very difficult. Second, the tasks may have different degrees of similarity in different dimensions, making that treating all dimensions with equal importance may be harmful to the KT process. To address these two challenges, this article proposes a novel orthogonal transfer (OT) method that is enabled by a cross-task mapping (CTM) strategy, which can achieve high-quality KT among heterogeneous tasks. For the first challenge, the CTM strategy maps the global best individual of one task from its original search space to the search space of the target task via an optimization process, which can handle the difference in task dimensionality. For the second challenge, the OT method is performed on the CTM-obtained individual and a random individual of the target task to find the best combination of different dimensions in these two individuals rather than treating all the dimensions equally, so as to achieve high-quality KT. To verify the effectiveness of the proposed OT method and the resulted OT-based multitask optimization (OTMTO) algorithm, this article not only uses the existing multitask optimization benchmark but also proposes a new benchmark test suite named multitask optimization problems (MTOPs) with DDs. Comprehensive experimental results on the existing and the proposed benchmarks show that the proposed OT method and the OTMTO algorithm are very advantageous in providing high-quality KT and in handling the heterogeneity of search space in MTOPs compared to the existing competitive evolutionary multitask optimization (EMTO) algorithms.

Neural architecture search using property guided synthesis

Neural architecture search (NAS) has become an increasingly important tool within the deep learning community in recent years, yielding many practical advancements in the design of deep neural network architectures. However, most existing approaches operate within highly structured design spaces, and hence (1) explore only a small fraction of the full search space of neural architectures while also (2) requiring significant manual effort from domain experts. In this work, we develop techniques that enable efficient NAS in a significantly larger design space. In particular, we propose to perform NAS in an abstract search space of program properties. Our key insights are as follows: (1) an abstract search space can be significantly smaller than the original search space, and (2) architectures with similar program properties should also have similar performance; thus, we can search more efficiently in the abstract search space. To enable this approach, we also introduce a novel efficient synthesis procedure, which performs the role of concretizing a set of promising program properties into a satisfying neural architecture. We implement our approach, αNAS, within an evolutionary framework, where the mutations are guided by the program properties. Starting with a ResNet-34 model, αNAS produces a model with slightly improved accuracy on CIFAR-10 but 96% fewer parameters. On ImageNet, αNAS is able to improve over Vision Transformer (30% fewer FLOPS and parameters), ResNet-50 (23% fewer FLOPS, 14% fewer parameters), and EfficientNet (7% fewer FLOPS and parameters) without any degradation in accuracy.

Towards One Shot Search Space Poisoning in Neural Architecture Search (Student Abstract)

We evaluate the robustness of a Neural Architecture Search (NAS) algorithm known as Efficient NAS (ENAS) against data agnostic poisoning attacks on the original search space with carefully designed ineffective operations. We empirically demonstrate how our one shot search space poisoning approach exploits design flaws in the ENAS controller to degrade predictive performance on classification tasks. With just two poisoning operations injected into the search space, we inflate prediction error rates for child networks upto 90% on the CIFAR-10 dataset.

A Progressive and Joint Method for Micro and Macro Architecture Search

Recently, neural architecture search (NAS) has achieved great success in design neural architectures automatically. Differentiable architecture search (DARTS) has succeeded in reducing computational cost and making the searching process efficient. Based on the cell-based search space in DARTS, we extend the search space by searching diverse cells at different stages and various connection between cells. We also use a progressive and joint method to search the micro and macro architecture together. Extensive experiments on CIFAR-10 demostrate that we can obtain a better result than the original search space.

Parallel Cooperative Coevolutionary Grey Wolf Optimizer for Path Planning Problem of Unmanned Aerial Vehicles.

The path planning of Unmanned Aerial Vehicles (UAVs) is a complex and hard task that can be formulated as a Large-Scale Global Optimization (LSGO) problem. A higher partition of the flight environment leads to an increase in route’s accuracy but at the expense of greater planning complexity. In this paper, a new Parallel Cooperative Coevolutionary Grey Wolf Optimizer (PCCGWO) is proposed to solve such a planning problem. The proposed PCCGWO metaheuristic applies cooperative coevolutionary concepts to ensure an efficient partition of the original search space into multiple sub-spaces with reduced dimensions. The decomposition of the decision variables vector into several sub-components is achieved and multi-swarms are created from the initial population. Each sub-swarm is then assigned to optimize a part of the LSGO problem. To form the complete solution, the representatives from each sub-swarm are combined. To reduce the computation time, an efficient parallel master-slave model is introduced in the proposed parameters-free PCCGWO. The master will be responsible for decomposing the original problem and constructing the context vector which contains the complete solution. Each slave is designed to evolve a sub-component and will send the best individual as its representative to the master after each evolutionary cycle. Demonstrative results show the effectiveness and superiority of the proposed PCCGWO-based planning technique in terms of several metrics of performance and nonparametric statistical analyses. These results show that the increase in the number of slaves leads to a more efficient result as well as a further improved computational time.

Search Reduction through Conservative Abstract-Space Based Heuristic

The efficiency of heuristic search depends dramatically on the quality of the heuristic function. For an optimal heuristic search, heuristics that estimate cost-to-goal better typically lead to faster searches. For a sub-optimal heuristic search such as weighted A*, the search speed depends more on the correlation between the heuristic and the true cost-to-goal. In this extended abstract, we discuss our preliminary work on computing heuristic functions that exploit this fact. In particular, we introduce a many-to-one mapping from an original search space to a conservative abstract space. Edges in the abstract space capture reachability among all corresponding nodes in the original space. We compute a heuristic in the conservative abstract space which when used by the search in the original space reduces the number of searched nodes. Our preliminary results on 3D navigation show that in more complex scenarios the speedup can be dramatic.

Solving Large-Scale Multiobjective Optimization Problems With Sparse Optimal Solutions via Unsupervised Neural Networks.

Due to the curse of dimensionality of search space, it is extremely difficult for evolutionary algorithms to approximate the optimal solutions of large-scale multiobjective optimization problems (LMOPs) by using a limited budget of evaluations. If the Pareto-optimal subspace is approximated during the evolutionary process, the search space can be reduced and the difficulty encountered by evolutionary algorithms can be highly alleviated. Following the above idea, this article proposes an evolutionary algorithm to solve sparse LMOPs by learning the Pareto-optimal subspace. The proposed algorithm uses two unsupervised neural networks, a restricted Boltzmann machine, and a denoising autoencoder to learn a sparse distribution and a compact representation of the decision variables, where the combination of the learnt sparse distribution and compact representation is regarded as an approximation of the Pareto-optimal subspace. The genetic operators are conducted in the learnt subspace, and the resultant offspring solutions then can be mapped back to the original search space by the two neural networks. According to the experimental results on eight benchmark problems and eight real-world problems, the proposed algorithm can effectively solve sparse LMOPs with 10000 decision variables by only 100000 evaluations.

Improving One-Shot NAS with Shrinking-and-Expanding Supernet

Training a supernet using a copy of shared weights has become a popular approach to speed up neural architecture search (NAS). However, it is difficult for supernet to accurately evaluate on a large-scale search space due to high weight coupling in weight-sharing setting. To address this, we present a shrinking-and-expanding supernet that decouples the shared parameters by reducing the degree of weight sharing, avoiding unstable and inaccurate performance estimation as in previous methods. Specifically, we propose a new shrinking strategy that progressively simplifies the original search space by discarding unpromising operators in a smart way. Based on this, we further present an expanding strategy by appropriately increasing parameters of the shrunk supernet. We provide comprehensive evidences showing that, in weight-sharing supernet, the proposed method SE-NAS brings more accurate and more stable performance estimation. Experimental results on ImageNet dataset indicate that SE-NAS achieves higher Top-1 accuracy than its counterparts under the same complexity constraint and search space. The ablation study is presented to further understand SE-NAS.

An Effective Variable Transformation Strategy in Multitasking Evolutionary Algorithms

Multitasking evolutionary algorithm (MTEA), which solves multiple optimization tasks simultaneously in a single run, has received considerable attention in the community of evolutionary computation, and several algorithms have been proposed in the literature. Unfortunately, knowledge transfer between constituent tasks may cause negative effect on algorithm performance, especially when the optimal solutions of all tasks are in different locations of the unified search space. To address this issue, an effective variable transformation strategy and the corresponding inverse transformation are proposed in multitasking optimization scenario. After using variable transformation strategy, the estimated optimal solutions of all tasks are both near the center point of the unified search space. More importantly, this strategy can enhance the task similarity, and then the effectiveness of knowledge transfer will probably be positive in this case, which can help us to improve the algorithm performance. Keeping this in mind, a multitasking evolutionary algorithm (named MTDE-VT) is realized as an instance by embedding the proposed variable transformation strategy into multitasking differential evolution. In MTDE-VT, the individuals in the original population are first transformed into new locations by the variable transformation strategy. Once the offspring is generated in the transformed unified search space, it must be transformed back to the original unified search space. The statistical analysis of experimental results on some multitasking optimization benchmark problems illustrates the superiority of the proposed MTDE-VT algorithm in terms of solution accuracy and robustness. Furthermore, the basic principle and the good parameter combination are also provided based on massive simulated data.

Trading Convergence Rate with Computational Budget in High Dimensional Bayesian Optimization

Scaling Bayesian optimisation (BO) to high-dimensional search spaces is a active and open research problems particularly when no assumptions are made on function structure. The main reason is that at each iteration, BO requires to find global maximisation of acquisition function, which itself is a non-convex optimization problem in the original search space. With growing dimensions, the computational budget for this maximisation gets increasingly short leading to inaccurate solution of the maximisation. This inaccuracy adversely affects both the convergence and the efficiency of BO. We propose a novel approach where the acquisition function only requires maximisation on a discrete set of low dimensional subspaces embedded in the original high-dimensional search space. Our method is free of any low dimensional structure assumption on the function unlike many recent high-dimensional BO methods. Optimising acquisition function in low dimensional subspaces allows our method to obtain accurate solutions within limited computational budget. We show that in spite of this convenience, our algorithm remains convergent. In particular, cumulative regret of our algorithm only grows sub-linearly with the number of iterations. More importantly, as evident from our regret bounds, our algorithm provides a way to trade the convergence rate with the number of subspaces used in the optimisation. Finally, when the number of subspaces is "sufficiently large", our algorithm's cumulative regret is at most O*(√TγT) as opposed to O*(√DTγT) for the GP-UCB of Srinivas et al. (2012), reducing a crucial factor √D where D being the dimensional number of input space. We perform empirical experiments to evaluate our method extensively, showing that its sample efficiency is better than the existing methods for many optimisation problems involving dimensions up to 5000.

Variable-grouping-based exponential crossover for differential evolution algorithm

The performance of differential evolution (DE) algorithm largely depends on its crossover operator, whose substantive characteristics are to make the algorithm search in a subspace of the original search space. Different crossover operators use different subspace divisions, and how to choose a suitable crossover operator for a specific optimisation problem is still an open issue. This paper proposes variable-grouping-based exponential crossover (VGExp), where all variables are divided into multiple groups based on interaction information, and the variables that are mutated simultaneously have a high probability of coming from the same group. Moreover, the solutions can improve the accuracy of the variable grouping and provide initial guidance for optimisation. Therefore, the proposed VGExp seamlessly combines variables grouping technique and differential evolution. The experiment results based on 30 CEC2014 test problems show that VGExp can improve the performance of most DE variants, and it is also better than other well-developed crossover operators.

A new approach using fuzzy DEA models to reduce search space and eliminate replications in simulation optimization problems

This article proposes a new combination of methods to increase optimization simulation efficiency and reliability, utilizing orthogonal arrays, fuzzy-data envelopment analysis (FDEA) with linear membership function, and discrete event simulation (DES). Considering a simulation optimization problem, experimental matrices are generated using orthogonal arrays and which simulation runs (scenarios) will be executed are defined, followed by FDEA to analyze and rank the scenarios in terms of their efficiency (considering occurrence of uncertainty). In this way, it is possible to reduce the search space of scenarios to be simulated, and avoid the need for replications in DES, without impairing the quality of the final solution. Six real cases that were solved by the proposed approach are presented. In order to highlight the efficiency of the proposed method, in Cases 5 and 6, all viable solutions of each of these problems were tested, ie, 100% of the search space was analyzed, and it was found that the solution obtained by the new method was statistically equal to the overall optimal solution. Note that for the other real cases solved, the solutions obtained by the proposed method were also statistically equal to those obtained from the original search space, and that analyzing 100% of the viable solutions space would be computationally impossible or impractical. These results confirmed the reliability and applicability of the proposed method, since it enabled a significant reduction in the search space for the simulation application compared to conventional simulation optimization techniques.

Low-Dimensional Euclidean Embedding for Visualization of Search Spaces in Combinatorial Optimization

This paper proposes a method for visualizing combinatorial search spaces named low-dimensional Euclidean embedding (LDEE). The proposed method transforms the original search space, such as a set of permutations or binary vectors, to $\pmb {\mathbb {R}}^{k}$ (with ${k} {=} {2}$ or 3 in practice) while aiming to preserve spatial relationships existing in the original space. The LDEE method uses the t-distributed stochastic neighbor embedding (t-SNE) to transform solutions from the original search space to the Euclidean one. In this paper, it is mathematically shown that the assumptions underlying the t-SNE method are valid in the case of permutation spaces with the Mallows distribution. The same is true for other metric spaces provided that the distribution of points is assumed to be normal with respect to the adopted metric. The embedding obtained using t-SNE is further refined to ensure visual separation of individual solutions. The visualization obtained using the LDEE method can be used for analyzing the behavior of the population in a population-based metaheuristic, the working of the genetic operators, etc. Examples of visualizations obtained using this method for the four peaks problem, the firefighter problem, the knapsack problem, the quadratic assignment problem, and the traveling salesman problem are presented in this paper.