Neural Architecture Search Research Articles

Scalable CNN-based classification of selective sweeps using derived allele frequencies.

Selective sweeps can successfully be distinguished from neutral genetic data using summary statistics and likelihood-based methods that analyze single nucleotide polymorphisms (SNPs). However, these methods are sensitive to confounding factors, such as severe population bottlenecks and old migration. By virtue of machine learning, and specifically convolutional neural networks (CNNs), new accurate classification models that are robust to confounding factors have been recently proposed. However, such methods are more computationally expensive than summary-statistic-based ones, yielding them impractical for processing large-scale genomic data. Moreover, SNP data are frequently preprocessed to improve classification accuracy, further exacerbating the long analysis times. To this end, we propose a 1D CNN-based model, dubbed FAST-NN, that does not require any preprocessing while using only derived allele frequencies instead of summary statistics or raw SNP data, thereby yielding a sample-size-invariant, scalable solution. We evaluated several data fusion approaches to account for the variance of the density of genetic diversity across genomic regions (a selective sweep signature), and performed an extensive neural architecture search based on a state-of-the-art reference network architecture (SweepNet). The resulting model, FAST-NN, outperforms the reference architecture by up to 12% inference accuracy over all challenging evolutionary scenarios with confounding factors that were evaluated. Moreover, FAST-NN is between 30× and 259× faster on a single CPU core, and between 2.0× and 6.2× faster on a GPU, when processing sample sizes between 128 and 1000 samples. Our work paves the way for the practical use of CNNs in large-scale selective sweep detection. https://github.com/SjoerdvandenBelt/FAST-NN.

Neural architecture search for image super-resolution: A review on the emerging state-of-the-art

Nowadays, complex and expensive neural architectures are seen by many as a way to improve the performance of existing models in image recognition, voice recognition, translation, and other tasks. Such a perspective caused an increased interest in expert architecture engineering within Deep Learning. Fueled by this interest, neural architecture search originated as a promising way to automate the tedious process of constructing a deep neural network by hand. Over the last five years, we have seen an increasing number of works focusing all efforts on studying the impact of automating deep neural network design. The spotlight has recently turned from automatically discovering classification models to other more complex tasks. Motivated by a desire for high-resolution images in real-world user-centered and expert computer vision applications, architecture search for super-resolution image restoration centers in approaches capable of automatically finding efficient and well-performing models. Here, we present a survey that, beyond delving into an overview of modern approaches to automatic neural network design, focuses on the recollection and study of neural architecture search approaches that have directed their efforts at the super-resolution image restoration tasks and future lines of research found within this emerging area of study.

Wind power forecasting based on ensemble deep learning with surrogate-assisted evolutionary neural architecture search and many-objective federated learning

Deep neural networks (DNN) have been widely used in wind power forecasting (WPF), however, they still encounter difficulties such as delayed variable selection, DNN architecture design, and network weight optimization. Therefore, an ensemble cascaded DNN (ECDNN) with surrogate-assisted evolutionary neural architecture search (SEANS) and many-objective federated learning (MOFL) is proposed and referred to as ECDNN-SENAS-MOFL. SEANS is proposed to determine the delayed variables and network architecture efficiently, which allows significantly enhancing the evolutionary search efficiency than traditional evolutionary neural architecture search (ENAS) approaches by employing a surrogate model to achieve fast fitness evaluations. Meanwhile, MOFL is developed by embedding the federated learning into many-objective optimization to obtain more reliable network weights based on the distributed modeling data than traditional centralized learning and federated learning, as well as solving the problems of data privacy protection and data islands. Furthermore, selective ensemble learning is introduced to build high-performance ensemble WPF model by combining the merits of multiple highly influential Pareto solutions. The effectiveness and superiority of the proposed ECDNN-SENAS-MOFL method are verified using an actual wind power dataset. The application results show that the proposed method can provide new insights into data-driven WPF modeling and has a great potential for practical applications.

Multi-objective evolutionary neural architecture search for network intrusion detection

Network Intrusion Detection (NID) becomes significantly important for protecting the security of information systems, as the frequency and complexity of network attacks are increasing with the rapid development of the Internet. Recent research studies have proposed various neural network models for NID, but they need to manually design the network architectures based on expert knowledge, which is very time-consuming. To solve this problem, this paper proposes a Multi-objective Evolutionary Neural Architecture Search (MENAS) method, which can automatically design neural network models for NID. First, a comprehensive search space is designed and then a weight-sharing mechanism is used to construct a supernet for NID, allowing each subnet to inherit weights from the supernet for direct performance evaluation. Subsequently, the subnets are encoded as chromosomes for multi-objective evolutionary search, which simultaneously optimizes two objectives: enhancing the model’s detection performance and reducing its complexity. To improve the search capability, a path-based crossover method is designed, which can iteratively refine the subnets’ architectures by simultaneously optimizing their accuracy and complexity for NID. At last, our MENAS method has been validated through extensive experiments on three well-known NID datasets: NSL-KDD, UNSW-NB15, and CICIDS2017. The experiments show that our MENAS method obtains an average 1.45% improvement on accuracy and an average 68.70% reduction on floating-point operations through multi-objective optimization process on six scenarios, which outperforms some state-of-the-art NID methods.

Semi-supervised accuracy predictor-based multi-objective neural architecture search

The rise of neural architecture search (NAS) demonstrates the deep exploration between the neural network architecture and its performance (e.g., accuracy). Many NAS methods are inefficient because they train all candidates from scratch to obtain their accuracies. Although predictor-based NAS algorithms have been vigorously developed to efficiently and accurately evaluate the performance of candidate architectures, the training of accuracy predictors still require hundreds of architectures with ground truth. To overcome this shortcoming, this paper investigates an evolutionary-based NAS method, which constructs a semi-supervised accuracy predictor to efficiently and accurately evaluate candidate architectures. A one-time extractor and strong regressors are implemented to further enhance the prediction performance of the semi-supervised accuracy predictor. Furthermore, a multi-objective approach is developed to find architectures with high ground truth in a tradeoff between high prediction accuracy and prediction confidence. Experimental results demonstrate the strong competitiveness of the proposed approach on NAS benchmarks. The code is available at https://github.com/outofstyle/SAPMNAS.

Multi-indicator based multi-objective evolutionary algorithm with application to neural architecture search

Multi-indicator based multi-objective evolutionary algorithm with application to neural architecture search

Noise-Disruption-Inspired Neural Architecture Search with Spatial–Spectral Attention for Hyperspectral Image Classification

In view of the complexity and diversity of hyperspectral images (HSIs), the classification task has been a major challenge in the field of remote sensing image processing. Hyperspectral classification (HSIC) methods based on neural architecture search (NAS) is a current attractive frontier that not only automatically searches for neural network architectures best suited to the characteristics of HSI data, but also avoids the possible limitations of manual design of neural networks when dealing with new classification tasks. However, the existing NAS-based HSIC methods have the following limitations: (1) the search space lacks efficient convolution operators that can fully extract discriminative spatial–spectral features, and (2) NAS based on traditional differentiable architecture search (DARTS) has performance collapse caused by unfair competition. To overcome these limitations, we proposed a neural architecture search method with receptive field spatial–spectral attention (RFSS-NAS), which is specifically designed to automatically search the optimal architecture for HSIC. Considering the core needs of the model in extracting more discriminative spatial–spectral features, we designed a novel and efficient attention search space. The core component of this innovative space is the receptive field spatial–spectral attention convolution operator, which is capable of precisely focusing on the critical information in the image, thus greatly enhancing the quality of feature extraction. Meanwhile, for the purpose of solving the unfair competition issue in the traditional differentiable architecture search (DARTS) strategy, we skillfully introduce the Noisy-DARTS strategy. The strategy ensures the fairness and efficiency of the search process and effectively avoids the risk of performance crash. In addition, to further improve the robustness of the model and ability to recognize difficult-to-classify samples, we proposed a fusion loss function by combining the advantages of the label smoothing loss and the polynomial expansion perspective loss function, which not only smooths the label distribution and reduces the risk of overfitting, but also effectively handles those difficult-to-classify samples, thus improving the overall classification accuracy. Experiments on three public datasets fully validate the superior performance of RFSS-NAS.

Comparative Analysis of Solar Radiation Forecasting Techniques in Zacatecas, Mexico

This work explores the prediction of daily Global Horizontal Irradiance (GHI) patterns in the region of Zacatecas, Mexico, using a diverse range of predictive models, encompassing traditional regressors and advanced neural networks like Evolutionary Neural Architecture Search (ENAS), Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), and Meta’s Prophet. This work addressing a notable gap in regional research, and aims to democratize access to accurate solar radiation forecasting methodologies. The evaluations carried out using the time series data obtained by Comisión Nacional del Agua (Conagua) covering the period from 2015 to 2018 reveal different performances of the model in different sky conditions, showcasing strengths in forecasting clear and partially cloudy days while encountering challenges with cloudy conditions. Overall, correlation coefficients (r) ranged between 0.55 and 0.72, with Root Mean Square Error % (RMSE %) values spanning from 20.05% to 20.54%, indicating moderate to good predictive accuracy. This study underscores the need for longer datasets to bolster future predictive capabilities. By democratizing access to these predictive tools, this research facilitates informed decision-making in renewable energy planning and sustainable development strategies tailored to the unique environmental dynamics of the region of Zacatecas and comparable regions.

RobMedNAS: searching robust neural network architectures for medical image synthesis

Investigating U-Net model robustness in medical image synthesis against adversarial perturbations, this study introduces RobMedNAS, a neural architecture search strategy for identifying resilient U-Net configurations. Through retrospective analysis of synthesized CT from MRI data, employing Dice coefficient and mean absolute error metrics across critical anatomical areas, the study evaluates traditional U-Net models and RobMedNAS-optimized models under adversarial attacks. Findings demonstrate RobMedNAS’s efficacy in enhancing U-Net resilience without compromising on accuracy, proposing a novel pathway for robust medical image processing.

Zero-Shot Proxy with Incorporated-Score for Lightweight Deep Neural Architecture Search

Zero-Shot Proxy with Incorporated-Score for Lightweight Deep Neural Architecture Search

Depth-adaptive graph neural architecture search for graph classification

In recent years, graph neural networks (GNNs) based on neighborhood aggregation schemes have become a promising method in various graph-based applications. To solve the expert-dependent and time-consuming problem in human-designed GNN architectures, graph neural architecture search (GNAS) has been popular. However, as the mainstream GNAS methods automatically design GNN architectures with fixed GNN depth, they cannot mine the true potential of GNN architectures for graph classification. Although a few GNAS methods have explored the importance of adaptive GNN depth based on fixed GNN architectures, they have not designed a general search space for graph classification, which limits the discovery of excellent GNN architectures. In this paper, we propose Depth-Adaptive Graph Neural Architecture Search for Graph Classification (DAGC), which systemically constructs and explores the search space for graph classification, rather than studying individual designs. Through decoupling the graph classification process, DAGC proposes a complete and flexible search space, including GNN depth, aggregation function, and pooling operation components. To this end, DAGC adopts a learnable agent based on reinforcement learning to effectively guide the search for depth-adaptive GNN architectures. Extensive experiments on five real-world datasets demonstrate that DAGC outperforms the state-of-the-art human-designed GNN architectures and GNAS methods. The code is available at: https://github.com/Zhen-Peng-Wu/DAGC.

PSO-based lightweight neural architecture search for object detection

In recent years, Neural Architecture Search (NAS) has received widespread attention for its remarkable ability to design neural networks. However, existing NAS works have mainly focused on network search, with limited emphasis on downstream applications after discovering efficient neural networks. In this paper, we propose a lightweight search strategy based on the particle swarm optimization algorithm and apply the searched network as backbone for object detection tasks. Specifically, we design a lightweight search space based on Ghostconv modules and improved Mobileblocks, achieving comprehensive exploration within the search space using variable-length encoding strategy. During the search process, to balance network performance and resource consumption, we propose a multi-objective fitness function and incorporated the classification accuracy, parameter size, and FLOPs of candidate individuals into optimization. For particle performance evaluation, we propose a new strategy based on weight sharing and dynamic early stopping, significantly accelerating the search process. Finally, we fine-tune the globally optimal particle decoded as the backbone, adding Ghost PAN feature fusion modules and detection heads to build an object detection model, and we achieve a 17.01% mAP on the VisDrone2019 dataset. Experimental results demonstrate the competitiveness of our algorithm in terms of search time and the balance between accuracy and efficiency, and also confirm the effectiveness of object detection models designed through NAS methods.

Advances in neural architecture search.

Automated machine learning (AutoML) has achieved remarkable success in automating the non-trivial process of designing machine learning models. Among the focal areas of AutoML, neural architecture search (NAS) stands out, aiming to systematically explore the complex architecture space to discover the optimal neural architecture configurations without intensive manual interventions. NAS has demonstrated its capability of dramatic performance improvement across a large number of real-world tasks. The core components in NAS methodologies normally include (i) defining the appropriate search space, (ii) designing the right search strategy and (iii) developing the effective evaluation mechanism. Although early NAS endeavors are characterized via groundbreaking architecture designs, the imposed exorbitant computational demands prompt a shift towards more efficient paradigms such as weight sharing and evaluation estimation, etc. Concurrently, the introduction of specialized benchmarks has paved the way for standardized comparisons of NAS techniques. Notably, the adaptability of NAS is evidenced by its capability of extending to diverse datasets, including graphs, tabular data and videos, etc., each of which requires a tailored configuration. This paper delves into the multifaceted aspects of NAS, elaborating on its recent advances, applications, tools, benchmarks and prospective research directions.

Neural Network Models for Ionospheric Electron Density Prediction at a Fixed Altitude Using Neural Architecture Search

AbstractSpecification and forecast of ionospheric parameters, such as ionospheric electron density (Ne), have been an important topic in space weather and ionospheric research. Neural networks (NNs) emerge as a powerful modeling tool for Ne prediction. However, heavy manual adjustments are time consuming to determine the optimal NN structures. In this work, we propose to use neural architecture search (NAS), an automatic machine learning method, to mitigate this problem. NAS aims to find the optimal network structure through the alternate optimization of the hyperparameters and the corresponding network parameters within a pre‐defined hyperparameter search space. A total of 16‐year data from Millstone Hill incoherent scatter radar (ISR) are used for the NN models. One single‐layer NN (SLNN) model and one deep NN (DNN) model are both trained with NAS, namely SLNN‐NAS and DNN‐NAS, for Ne prediction and compared with their manually tuned counterparts (SLNN and DNN) based on previous studies. Our results show that SLNN‐NAS and DNN‐NAS outperformed SLNN and DNN, respectively. These NN predictions of Ne daily variation patterns reveal a 27‐day mid‐latitude topside Ne variation, which cannot be reasonably represented by traditional empirical models developed using monthly averages. DNN‐NAS yields the best prediction accuracy measured by quantitative metrics and rankings of daily pattern prediction, especially with an improvement in mean absolute error more than 10% compared to the SLNN model. The limited improvement of NAS is likely due to the network complexity and the limitation of fully connected NN without the time histories of input parameters.

Redefining Neural Architecture Search of Heterogeneous Multinetwork Models by Characterizing Variation Operators and Model Components.

With neural architecture search (NAS) methods gaining ground on manually designed deep neural networks-even more rapidly as model sophistication escalates-the research trend is shifting toward arranging different and often increasingly complex NAS spaces. In this conjuncture, delineating algorithms which can efficiently explore these search spaces can result in a significant improvement over currently used methods, which, in general, randomly select the structural variation operator, hoping for a performance gain. In this article, we investigate the effect of different variation operators in a complex domain, that of multinetwork heterogeneous neural models. These models have an extensive and complex search space of structures as they require multiple subnetworks within the general model in order to answer different output types. From that investigation, we extract a set of general guidelines whose application is not limited to that particular type of model and are useful to determine the direction in which an architecture optimization method could find the largest improvement. To deduce the set of guidelines, we characterize both the variation operators, according to their effect on the complexity and performance of the model; and the models, relying on diverse metrics which estimate the quality of the different parts composing it.

Combining Compressed Sensing and Neural Architecture Search for Sensor-Near Vibration Diagnostics

Combining Compressed Sensing and Neural Architecture Search for Sensor-Near Vibration Diagnostics

AutoCTS++: zero-shot joint neural architecture and hyperparameter search for correlated time series forecasting

AutoCTS++: zero-shot joint neural architecture and hyperparameter search for correlated time series forecasting

SalNAS: Efficient Saliency-prediction Neural Architecture Search with self-knowledge distillation

Recent advancements in deep convolutional neural networks have significantly improved the performance of saliency prediction. However, the manual configuration of the neural network architectures requires domain knowledge expertise and can still be time-consuming and error-prone. To solve this, we propose a new Neural Architecture Search (NAS) framework for saliency prediction with two contributions. Firstly, a supernet for saliency prediction is built with a weight-sharing network containing all candidate architectures, by integrating a dynamic convolution into the encoder–decoder in the supernet, termed SalNAS. Secondly, despite the fact that SalNAS is highly efficient (20.98 million parameters), it can suffer from the lack of generalization. To solve this, we propose a self-knowledge distillation approach, termed Self-KD, that trains the student SalNAS with the weighted average information between the ground truth and the prediction from the teacher model. The teacher model, while sharing the same architecture, contains the best-performing weights chosen by cross-validation. Self-KD can generalize well without the need to compute the gradient in the teacher model, enabling an efficient training system. By utilizing Self-KD, SalNAS outperforms other state-of-the-art saliency prediction models in most evaluation rubrics across seven benchmark datasets while being a lightweight model. The code will be available at https://github.com/chakkritte/SalNAS.

Multi-objective meta-learning

Meta-learning has arisen as a powerful tool for many machine learning problems. With multiple factors to be considered when designing learning models for real-world applications, meta-learning with multiple objectives has attracted much attention recently. However, existing works either linearly combine multiple objectives into one objective or adopt evolutionary algorithms to handle it, where the former approach needs to pay high computational cost to tune the combination coefficients while the latter approach is computationally heavy and incapable to be integrated into gradient-based optimization. To alleviate those limitations, in this paper, we aim to propose a generic gradient-based Multi-Objective Meta-Learning (MOML) framework with applications in many machine learning problems. Specifically, the MOML framework formulates the objective function of meta-learning with multiple objectives as a Multi-Objective Bi-Level optimization Problem (MOBLP) where the upper-level subproblem is to solve several possibly conflicting objectives for the meta-learner. Different from those existing works, in this paper, we propose a gradient-based algorithm to solve the MOBLP. Specifically, we devise the first gradient-based optimization algorithm by alternately solving the lower-level and upper-level subproblems via the gradient descent method and the gradient-based multi-objective optimization method, respectively. Theoretically, we prove the convergence property and provide a non-asymptotic analysis of the proposed gradient-based optimization algorithm. Empirically, extensive experiments justify our theoretical results and demonstrate the superiority of the proposed MOML framework for different learning problems, including few-shot learning, domain adaptation, multi-task learning, neural architecture search, and reinforcement learning. The source code of MOML is available at https://github.com/Baijiong-Lin/MOML.

Towards efficient AutoML: a pipeline synthesis approach leveraging pre-trained transformers for multimodal data

This paper introduces an Automated Machine Learning (AutoML) framework specifically designed to efficiently synthesize end-to-end multimodal machine learning pipelines. Traditional reliance on the computationally demanding Neural Architecture Search is minimized through the strategic integration of pre-trained transformer models. This innovative approach enables the effective unification of diverse data modalities into high-dimensional embeddings, streamlining the pipeline development process. We leverage an advanced Bayesian Optimization strategy, informed by meta-learning, to facilitate the warm-starting of the pipeline synthesis, thereby enhancing computational efficiency. Our methodology demonstrates its potential to create advanced and custom multimodal pipelines within limited computational resources. Extensive testing across 23 varied multimodal datasets indicates the promise and utility of our framework in diverse scenarios. The results contribute to the ongoing efforts in the AutoML field, suggesting new possibilities for efficiently handling complex multimodal data. This research represents a step towards developing more efficient and versatile tools in multimodal machine learning pipeline development, acknowledging the collaborative and ever-evolving nature of this field.