Neural Architecture Search Method Research Articles

Model optimization and acceleration method based on meta-learning and model pruning for laser vision weld tracking system

Purpose This paper aims to propose a lightweight, high-accuracy object detection model designed to enhance seam tracking quality under strong arcs and splashes condition. Simultaneously, the model aims to reduce computational costs. Design/methodology/approach The lightweight model is constructed based on Single Shot Multibox Detector (SSD). First, a neural architecture search method based on meta-learning and genetic algorithm is introduced to optimize pruning strategy, reducing human intervention and improving efficiency. Additionally, the Alternating Direction Method of Multipliers (ADMM) is used to perform structural pruning on SSD, effectively compressing the model with minimal loss of accuracy. Findings Compared to state-of-the-art models, this method better balances feature extraction accuracy and inference speed. Furthermore, seam tracking experiments on this welding robot experimental platform demonstrate that the proposed method exhibits excellent accuracy and robustness in practical applications. Originality/value This paper presents an innovative approach that combines ADMM structural pruning and meta-learning-based neural architecture search to significantly enhance the efficiency and performance of the SSD network. This method reduces computational cost while ensuring high detection accuracy, providing a reliable solution for welding robot laser vision systems in practical applications.

Optimizing graph neural network architectures for schizophrenia spectrum disorder prediction using evolutionary algorithms

Background and Objective:The accurate diagnosis of schizophrenia spectrum disorder plays an important role in improving patient outcomes, enabling timely interventions, and optimizing treatment plans. Functional connectivity analysis, utilizing functional magnetic resonance imaging data, has been demonstrated to offer invaluable biomarkers conducive to clinical diagnosis. However, previous studies mainly focus on traditional machine learning methods or hand-crafted neural networks, which may not fully capture the spatial topological relationship between brain regions. Methods:This paper proposes an evolutionary algorithm (EA) based graph neural architecture search (GNAS) method. EA-GNAS has the ability to search for high-performance graph neural networks for schizophrenia spectrum disorder prediction. Moreover, we adopt GNNExplainer to investigate the explainability of the acquired architectures, ensuring that the model’s predictions are both accurate and comprehensible. Results:The results suggest that the graph neural network model, derived using genetic algorithm search, outperforms under five-fold cross-validation, achieving a fitness of 0.1850. Relative to conventional machine learning and other deep learning approaches, the proposed method yields superior accuracy, F1 score, and AUC values of 0.8246, 0.8438, and 0.8258, respectively. Conclusion:Based on a multi-site dataset from schizophrenia spectrum disorder patients, the findings reveal an enhancement over prior methods, advancing our comprehension of brain function and potentially offering a biomarker for diagnosing schizophrenia spectrum disorder.

DIPO: Differentiable Parallel Operation Blocks for Surgical Neural Architecture Search.

Deep learning has been used across a large number of computer vision tasks, however designing the network architectures for each task is time consuming. Neural Architecture Search (NAS) promises to automatically build neural networks, optimised for the given task and dataset. However, most NAS methods are constrained to a specific macro-architecture design which makes it hard to apply to different tasks (classification, detection, segmentation). Following the work in Differentiable NAS (DNAS), we present a simple and efficient NAS method, Differentiable Parallel Operation (DIPO), that constructs a local search space in the form of a DIPO block, and can easily be applied to any convolutional network by injecting it in-place of the convolutions. The DIPO block's internal architecture and parameters are automatically optimised end-to-end for each task. We demonstrate the flexibility of our approach by applying DIPO to 4 model architectures (U-Net, HRNET, KAPAO and YOLOX) across different surgical tasks (surgical scene segmentation, surgical instrument detection, and surgical instrument pose estimation) and evaluated across 5 datasets. Results show significant improvements in surgical scene segmentation (+10.5% in CholecSeg8K, +13.2% in CaDIS), instrument detection (+1.5% in ROBUST-MIS, +5.3% in RoboKP), and instrument pose estimation (+9.8% in RoboKP).

TrajectoryNAS: A Neural Architecture Search for Trajectory Prediction.

Autonomous driving systems are a rapidly evolving technology. Trajectory prediction is a critical component of autonomous driving systems that enables safe navigation by anticipating the movement of surrounding objects. Lidar point-cloud data provide a 3D view of solid objects surrounding the ego-vehicle. Hence, trajectory prediction using Lidar point-cloud data performs better than 2D RGB cameras due to providing the distance between the target object and the ego-vehicle. However, processing point-cloud data is a costly and complicated process, and state-of-the-art 3D trajectory predictions using point-cloud data suffer from slow and erroneous predictions. State-of-the-art trajectory prediction approaches suffer from handcrafted and inefficient architectures, which can lead to low accuracy and suboptimal inference times. Neural architecture search (NAS) is a method proposed to optimize neural network models by using search algorithms to redesign architectures based on their performance and runtime. This paper introduces TrajectoryNAS, a novel neural architecture search (NAS) method designed to develop an efficient and more accurate LiDAR-based trajectory prediction model for predicting the trajectories of objects surrounding the ego vehicle. TrajectoryNAS systematically optimizes the architecture of an end-to-end trajectory prediction algorithm, incorporating all stacked components that are prerequisites for trajectory prediction, including object detection and object tracking, using metaheuristic algorithms. This approach addresses the neural architecture designs in each component of trajectory prediction, considering accuracy loss and the associated overhead latency. Our method introduces a novel multi-objective energy function that integrates accuracy and efficiency metrics, enabling the creation of a model that significantly outperforms existing approaches. Through empirical studies, TrajectoryNAS demonstrates its effectiveness in enhancing the performance of autonomous driving systems, marking a significant advancement in the field. Experimental results reveal that TrajcetoryNAS yields a minimum of 4.8 higger accuracy and 1.1* lower latency over competing methods on the NuScenes dataset.

Multi-objective evolutionary neural architecture search for network intrusion detection

Network Intrusion Detection (NID) becomes significantly important for protecting the security of information systems, as the frequency and complexity of network attacks are increasing with the rapid development of the Internet. Recent research studies have proposed various neural network models for NID, but they need to manually design the network architectures based on expert knowledge, which is very time-consuming. To solve this problem, this paper proposes a Multi-objective Evolutionary Neural Architecture Search (MENAS) method, which can automatically design neural network models for NID. First, a comprehensive search space is designed and then a weight-sharing mechanism is used to construct a supernet for NID, allowing each subnet to inherit weights from the supernet for direct performance evaluation. Subsequently, the subnets are encoded as chromosomes for multi-objective evolutionary search, which simultaneously optimizes two objectives: enhancing the model’s detection performance and reducing its complexity. To improve the search capability, a path-based crossover method is designed, which can iteratively refine the subnets’ architectures by simultaneously optimizing their accuracy and complexity for NID. At last, our MENAS method has been validated through extensive experiments on three well-known NID datasets: NSL-KDD, UNSW-NB15, and CICIDS2017. The experiments show that our MENAS method obtains an average 1.45% improvement on accuracy and an average 68.70% reduction on floating-point operations through multi-objective optimization process on six scenarios, which outperforms some state-of-the-art NID methods.

Semi-supervised accuracy predictor-based multi-objective neural architecture search

The rise of neural architecture search (NAS) demonstrates the deep exploration between the neural network architecture and its performance (e.g., accuracy). Many NAS methods are inefficient because they train all candidates from scratch to obtain their accuracies. Although predictor-based NAS algorithms have been vigorously developed to efficiently and accurately evaluate the performance of candidate architectures, the training of accuracy predictors still require hundreds of architectures with ground truth. To overcome this shortcoming, this paper investigates an evolutionary-based NAS method, which constructs a semi-supervised accuracy predictor to efficiently and accurately evaluate candidate architectures. A one-time extractor and strong regressors are implemented to further enhance the prediction performance of the semi-supervised accuracy predictor. Furthermore, a multi-objective approach is developed to find architectures with high ground truth in a tradeoff between high prediction accuracy and prediction confidence. Experimental results demonstrate the strong competitiveness of the proposed approach on NAS benchmarks. The code is available at https://github.com/outofstyle/SAPMNAS.

Noise-Disruption-Inspired Neural Architecture Search with Spatial–Spectral Attention for Hyperspectral Image Classification

In view of the complexity and diversity of hyperspectral images (HSIs), the classification task has been a major challenge in the field of remote sensing image processing. Hyperspectral classification (HSIC) methods based on neural architecture search (NAS) is a current attractive frontier that not only automatically searches for neural network architectures best suited to the characteristics of HSI data, but also avoids the possible limitations of manual design of neural networks when dealing with new classification tasks. However, the existing NAS-based HSIC methods have the following limitations: (1) the search space lacks efficient convolution operators that can fully extract discriminative spatial–spectral features, and (2) NAS based on traditional differentiable architecture search (DARTS) has performance collapse caused by unfair competition. To overcome these limitations, we proposed a neural architecture search method with receptive field spatial–spectral attention (RFSS-NAS), which is specifically designed to automatically search the optimal architecture for HSIC. Considering the core needs of the model in extracting more discriminative spatial–spectral features, we designed a novel and efficient attention search space. The core component of this innovative space is the receptive field spatial–spectral attention convolution operator, which is capable of precisely focusing on the critical information in the image, thus greatly enhancing the quality of feature extraction. Meanwhile, for the purpose of solving the unfair competition issue in the traditional differentiable architecture search (DARTS) strategy, we skillfully introduce the Noisy-DARTS strategy. The strategy ensures the fairness and efficiency of the search process and effectively avoids the risk of performance crash. In addition, to further improve the robustness of the model and ability to recognize difficult-to-classify samples, we proposed a fusion loss function by combining the advantages of the label smoothing loss and the polynomial expansion perspective loss function, which not only smooths the label distribution and reduces the risk of overfitting, but also effectively handles those difficult-to-classify samples, thus improving the overall classification accuracy. Experiments on three public datasets fully validate the superior performance of RFSS-NAS.

Modulation recognition of underwater acoustic communication signals based on neural architecture search

Modulation recognition of underwater communication signals is a critical aspect of underwater information confrontation. However, the current deep learning-based methods for underwater communication modulation recognition often mimic neural network architectures used in image processing and speech processing, tending to perform poorly in low signal-to-noise ratio (SNR) conditions. This paper introduces a novel approach by utilizing neural architecture search (NAS) method. By automatically searching for neural architectures that are well-suited for modulation recognition in underwater environment, our proposed method improves the classification performance particularly in low SNR conditions. Furthermore, we have also proposed a recognition method based on attention mechanism and feature fusion, which substantially enhances the accuracy of identifying phase-modulated signals. Numerical simulation and experimental data are used to demonstrate performance of proposed methods.

Multi-Objective Evolutionary Neural Architecture Search with Weight-Sharing Supernet

Deep neural networks have played a crucial role in the field of deep learning, achieving significant success in practical applications. The architecture of neural networks is key to their performance. In the past few years, these architectures have been manually designed by experts with rich domain knowledge. Additionally, the optimal neural network architecture can vary depending on specific tasks and data distributions. Neural Architecture Search (NAS) is a class of techniques aimed at automatically searching for and designing neural network architectures according to the given tasks and data. Specifically, evolutionary-computation-based NAS methods are known for their strong global search capability and have aroused widespread interest in recent years. Although evolutionary-computation-based NAS has achieved success in a wide range of research and applications, it still faces bottlenecks in training and evaluating a large number of individuals during optimization. In this study, we first devise a multi-objective evolutionary NAS framework based on a weight-sharing supernet to improve the search efficiency of traditional evolutionary-computation-based NAS. This framework combines the population optimization characteristic of evolutionary algorithms with the weight-sharing ideas in one-shot models. We then design a bi-population MOEA/D algorithm based on the proposed framework to effectively solve the NAS problem. By constructing two sub-populations with different optimization objectives, the algorithm can effectively explore network architectures of various sizes in complex search spaces. An inter-population communication mechanism further enhances the algorithm’s exploratory capability, enabling it to find network architectures with uniform distribution and high diversity. Finally, we conduct performance comparison experiments on image classification datasets of different scales and complexities. Experimental results demonstrate the effectiveness of the proposed multi-objective evolutionary NAS framework and the practicality and transferability of the introduced bi-population MOEA/D-based NAS method compared to existing state-of-the-art NAS methods.

Multiscale Feature Search-Based Graph Convolutional Network for Hyperspectral Image Classification

With the development of hyperspectral sensors, the availability of hyperspectral images (HSIs) has increased significantly, prompting advancements in deep learning-based hyperspectral image classification (HSIC) methods. Recently, graph convolutional networks (GCNs) have been proposed to process graph-structured data in non-Euclidean domains, and have been used for HSIC. The superpixel segmentation should be implemented first in the GCN-based methods, however, it is difficult to manually select the optimal superpixel segmentation sizes to obtain the useful information for classification. To solve this problem, we constructed a HSIC model based on a multiscale feature search-based graph convolutional network (MFSGCN) in this study. Firstly, pixel-level features of HSIs are extracted sequentially using 3D asymmetric decomposition convolution and 2D convolution. Then, superpixel-level features at different scales are extracted using multilayer GCNs. Finally, the neural architecture search (NAS) method is used to automatically assign different weights to different scales of superpixel features. Thus, a more discriminative feature map is obtained for classification. Compared with other GCN-based networks, the MFSGCN network can automatically capture features and obtain higher classification accuracy. The proposed MFSGCN model was implemented on three commonly used HSI datasets and compared to some state-of-the-art methods. The results confirm that MFSGCN effectively improves accuracy.

Automated federated learning‐based adversarial attack and defence in industrial control systems

AbstractWith the development of deep learning and federated learning (FL), federated intrusion detection systems (IDSs) based on deep learning have played a significant role in securing industrial control systems (ICSs). However, adversarial attacks on ICSs may compromise the ability of deep learning‐based IDSs to accurately detect cyberattacks, leading to serious consequences. Moreover, in the process of generating adversarial samples, the selection of replacement models lacks an effective method, which may not fully expose the vulnerabilities of the models. The authors first propose an automated FL‐based method to generate adversarial samples in ICSs, called AFL‐GAS, which uses the principle of transfer attack and fully considers the importance of replacement models during the process of adversarial sample generation. In the proposed AFL‐GAS method, a lightweight neural architecture search method is developed to find the optimised replacement model composed of a combination of four lightweight basic blocks. Then, to enhance the adversarial robustness, the authors propose a multi‐objective neural architecture search‐based IDS method against adversarial attacks in ICSs, called MoNAS‐IDSAA, by considering both classification performance on regular samples and adversarial robustness simultaneously. The experimental results on three widely used intrusion detection datasets in ICSs, such as secure water treatment (SWaT), Water Distribution, and Power System Attack, demonstrate that the proposed AFL‐GAS method has obvious advantages in evasion rate and lightweight compared with other four methods. Besides, the proposed MoNAS‐IDSAA method not only has a better classification performance, but also has obvious advantages in model adversarial robustness compared with one manually designed federated adversarial learning‐based IDS method.

Enhance audio-visual segmentation with hierarchical encoder and audio guidance

As one of the pivotal technologies leading towards embodied intelligence, audio-visual segmentation is geared towards achieving precise segmentation of sounding objects, offering vast application prospects in scenarios such as emergency rescue and natural exploration. Nevertheless, the performance of audio-visual segmentation technology encounters limitations stemming from challenges related to the adaptation and fusion of cross-modal information encoding, as well as the decoding and generation of masks. To address these issues, this paper explores the adaptation of multi-modal information based on a shared encoder by employing a neural architecture search method to design a hierarchical encoder cooperation module for enhanced information interaction. An intermediate loss is leveraged to help the encoder to keep spatial knowledge reserved. Furthermore, an audio-guided class-aware decoder is devised to guide the generation of masks. Our approach has yielded competitive experimental results across multiple datasets, thus substantiating its effectiveness.

DARTS-based morphological neural network design and application in bearing fault diagnosis

Morphological filters are widely used in data pre-processing, whereas the existing morphological filters are not optimal in terms of performance, since they are designed by combining the basic morphological operators together artificially. Hence, we proposed an automatic morphological neural network design method based on differentiable architecture search (DARTS), which is a neural architecture search (NAS) method. Firstly, the mechanism is revealed that the morphological dilation with flat structural elements behaves the same as Maxpooling without down-sampling. Secondly, morphological dilation and erosion neural network layers were defined. Thirdly, using DARTS and the search space with morphological neural network layers, end-to-end morphological neural network was implemented. Finally, the morphological neural network was successfully applied in bearing fault diagnosis. Experiments on the three datasets show that the proposed method improved the feature extraction capabilities of neural networks, then achieved end-to-end bearing fault diagnosis, and the diagnosis accuracy was improved by more than 2.0%.

Wavelet-driven differentiable architecture search for planetary gear fault diagnosis

With the advancement of artificial intelligence and the accumulation of industrial big data, intelligent diagnosis methods based on deep learning have become the mainstream for diagnosing mechanical faults in manufacturing systems. Despite this, developing high-performance neural network models for specific tasks necessitates a substantial amount of expert knowledge. This consumes a considerable amount of time in the trial-and-error process, thereby constraining the progress in neural network development. Neural architecture search (NAS) offers a solution to address this problem. However, the feature extraction operators in the existing NAS search space are primarily imported from other fields, leading to domain bias and a lack of interpretability. To address these challenges, we present a NAS method driven by wavelets. To be specific, the wavelet operators that can extract fault related features from vibration signals is added to the search space, and the gradient optimization strategy is utilized to search the optimal architecture from the hypernet. The effectiveness of the proposed method is validated through a dataset specific to planetary gears. Upon comparison with other models, it is evident that the proposed method exhibits superior performance across all added noise levels.

Multimodal Graph Neural Architecture Search under Distribution Shifts

Multimodal graph neural architecture search (MGNAS) has shown great success for automatically designing the optimal multimodal graph neural network (MGNN) architecture by leveraging multimodal representation, crossmodal information and graph structure in one unified framework. However, existing MGNAS fails to handle distribution shifts that naturally exist in multimodal graph data, since the searched architectures inevitably capture spurious statistical correlations under distribution shifts. To solve this problem, we propose a novel Out-of-distribution Generalized Multimodal Graph Neural Architecture Search (OMG-NAS) method which optimizes the MGNN architecture with respect to its performance on decorrelated OOD data. Specifically, we propose a multimodal graph representation decorrelation strategy, which encourages the searched MGNN model to output representations that eliminate spurious correlations through iteratively optimizing the feature weights and controller. In addition, we propose a global sample weight estimator that facilitates the sharing of optimal sample weights learned from existing architectures. This design promotes the effective estimation of the sample weights for candidate MGNN architectures to generate decorrelated multimodal graph representations, concentrating more on the truly predictive relations between invariant features and ground-truth labels. Extensive experiments on real-world multimodal graph datasets demonstrate the superiority of our proposed method over SOTA baselines.

An evolutionary neural architecture search method based on performance prediction and weight inheritance

Evolutionary Neural Architecture Search (ENAS) algorithms attract great attention since they can automatically search for appropriate network architectures for a given task. However, most ENAS algorithms suffer from a prohibitive computational burden. Moreover, some of these approaches directly use performance predictors for evaluations, which may introduce inaccurate assessments and harm the evolution. To overcome these shortcomings, we propose an efficient ENAS algorithm named EPPGA. EPPGA employs a predictor to pre-select potentially high-performing offspring, enhancing the performance and accelerating the evolution. As the offspring will be further accurately evaluated, even potentially inaccurate predictions will not adversely affect the evolution. Furthermore, a weight inheritance method is suggested to accelerate the evaluation, and new genetic operations are developed to produce offspring that share a substantial proportion of beneficial genetic materials with one parent, improving the performance predictor's effectiveness and promoting weight inheritance. Finally, a new efficient backbone block structure is designed to facilitate the search for lightweight networks. The experimental results demonstrate that EPPGA is a highly competitive algorithm on three benchmarks in terms of accuracy, model size, and computational cost, reveal the superiority of the proposed block structure, and confirm the effectiveness of the proposed performance predictor and weight inheritance method.

Neural architecture search via progressive partial connection with attention mechanism

Differentiable architecture search requires a larger computational consumption during architecture search, and there exists the depth gap problem under deeper network architecture. In this paper, we propose an attention-based progressive partially connected neural architecture search method (PPCAtt-NAS) to address these two issues. First, we introduce a progressive search strategy in the architecture search phase, build up the sophistication of the architecture gradually and perform path-level pruning in stages to bridge the depth gap. Second, we adopt a partial search scheme that performs channel-level partial sampling of the network architecture to further reduce the computational complexity of the architecture search. In addition, an attention mechanism is devised to improve the architecture search capability by enhancing the relevance between the feature channels. Finally, we conduct extensive comparison experiments with state-of-the-art methods on several public datasets, and our method is able to present higher architecture performance.

Efficient spiking neural network design via neural architecture search

Spiking neural networks (SNNs) are brain-inspired models that utilize discrete and sparse spikes to transmit information, thus having the property of energy efficiency. Recent advances in learning algorithms have greatly improved SNN performance due to the automation of feature engineering. While the choice of neural architecture plays a significant role in deep learning, the current SNN architectures are mainly designed manually, which is a time-consuming and error-prone process. In this paper, we propose a spiking neural architecture search (NAS) method that can automatically find efficient SNNs. To tackle the challenge of long search time faced by SNNs when utilizing NAS, the proposed NAS encodes candidate architectures in a branchless spiking supernet which significantly reduces the computation requirements in the search process. Considering that real-world tasks prefer efficient networks with optimal accuracy under a limited computational budget, we propose a Synaptic Operation (SynOps)-aware optimization to automatically find the computationally efficient subspace of the supernet. Experimental results show that, in less search time, our proposed NAS can find SNNs with higher accuracy and lower computational cost than state-of-the-art SNNs. We also conduct experiments to validate the search process and the trade-off between accuracy and computational cost.

Latency-Constrained Neural Architecture Search Method for Efficient Model Deployment on RISC-V Devices

The rapid development of the RISC-V instruction set architecture (ISA) has garnered significant attention in the realm of deep neural network applications. While hardware-aware neural architecture search (NAS) methods for ARM, X86, and GPUs have been extensively explored, research specifically targeting RISC-V remains limited. In light of this, we propose a latency-constrained NAS (LC-NAS) method specifically designed for RISC-V. This method enables efficient network searches without the requirement of network training. Concretely, in the training-free NAS framework, we introduce an RISC-V latency evaluation module that includes two implementations: a lookup table and a latency predictor based on a deep neural network. To obtain real latency data, we have designed a specialized data collection pipeline for RISC-V devices, which allows for precise end-to-end hardware latency measurements. We validate the effectiveness of our method in the NAS-Bench-201 search space. Experimental results demonstrate that our method can efficiently search for latency-constrained networks for RISC-V devices within seconds while maintaining high accuracy. Additionally, our method can easily integrate with existing training-free NAS approaches.

ReCNAS: Resource-Constrained Neural Architecture Search Based on Differentiable Annealing and Dynamic Pruning.

The differentiable neural architecture search (NAS) framework has obtained extensive attention and achieved remarkable performance due to its search efficiency. However, most existing differentiable NAS methods still suffer from issues of model collapse, degenerated search-evaluation correlation, and inefficient hardware deployment, which causes the searched architectures to be suboptimal in accuracy and cannot meet different computation resource constraints (e.g., FLOPs and latency). In this article, we propose a novel resource-constrained NAS (ReCNAS) method, which can efficiently search high-performance architectures that satisfy the given constraints, and deal with the issues observed in previous differentiable NAS methods from three aspects: search space, search strategy, and resource adaptability. First, we introduce an elastic densely connected layerwise search space, which decouples the architecture depth representation from the search of candidate operations to alleviate the aggregation of skip connections and architecture redundancies. Second, a scheme of group annealing and progressive pruning is proposed to improve the efficiency and bridge the search-evaluation gap, which steadily forces the architecture parameters close to binary distribution and progressively prunes the inferior operations. Third, we present a novel resource-constrained architecture generation method, which prunes the redundant channel throughout the search based on dynamic programming, making the searched architecture scalable to different devices and requirements. Extensive experimental results demonstrate the efficiency and search stability of our ReCNAS, which is capable of discovering high-performance architectures on different datasets and tasks, surpassing other NAS methods, while tightly meeting the target resource constraints without any tuning required. Besides, the searched architectures show strong generalizability to other complex vision tasks.