Neural Architecture Search Research Articles

Understanding and Accelerating Neural Architecture Search With Training-Free and Theory-Grounded Metrics

Understanding and Accelerating Neural Architecture Search With Training-Free and Theory-Grounded Metrics

Stereo matching of binocular laparoscopic images with improved densely connected neural architecture search.

Stereo matching is a crucial technology in the binocular laparoscopic-based surgical navigation systems. In recent years, neural networks have been widely applied to stereo matching and demonstrated outstanding performance. however, this method heavily relies on manual feature engineering meaning that professionals must be involved in the feature extraction and matching. This process is both time-consuming and demands specific expertise. This paper introduces a novel stereo matching framework DCStereo that realizes a fully automatic neural architecture design for the stereo matching of binocular laparoscopic images. The proposed framework utilizes a densely connected search space which enables a more flexible and diverse architecture composition. Furthermore, the proposed algorithm leverages the channel and path sampling strategies to reduce memory consumption during searching. Empirically, our searched DCStereo on the SCARED training dataset achieves a mean absolute error of 3.589mm on the test dataset, which outperforms hand-crafted stereo matching methods and other approaches. Furthermore, when directly testing on the SERV-CT dataset, our DCStereo demonstrates better generalization ability than other methods. Our proposed approach leverages the neural architecture search technique and a densely connected search space for automatic neural architecture design in stereo matching of binocular laparoscopic images. Our method delivers advanced performance on the SCARED dataset and promising results on the SERV-CT dataset. These findings demonstrate the potential of our approach for improving clinical surgical navigation systems.

Neural architecture search for radio map reconstruction with partially labeled data

In this paper, we tackle the challenging task of reconstructing Received Signal Strength (RSS) maps by harnessing location-dependent radio measurements and augmenting them with supplementary data related to the local environment. This side information includes city plans, terrain elevations, and the locations of gateways. The quantity of available supplementary data varies, necessitating the utilization of Neural Architecture Search (NAS) to tailor the neural network architecture to the specific characteristics of each setting. Our approach takes advantage of NAS’s adaptability, allowing it to automatically explore and pinpoint the optimal neural network architecture for each unique scenario. This adaptability ensures that the model is finely tuned to extract the most relevant features from the input data, thereby maximizing its ability to accurately reconstruct RSS maps. We demonstrate the effectiveness of our approach using three distinct datasets, each corresponding to a major city. Notably, we observe significant enhancements in areas near the gateways, where fluctuations in the mean received signal power are typically more pronounced. This underscores the importance of NAS-driven architectures in capturing subtle spatial variations. We also illustrate how NAS efficiently identifies the architecture of a Neural Network using both labeled and unlabeled data for Radio Map reconstruction. Our findings emphasize the potential of NAS as a potent tool for improving the precision and applicability of RSS map reconstruction techniques in urban environments.

Multi-label neural architecture search for chest radiography image classification

Multi-label neural architecture search for chest radiography image classification

Computer-aided diagnosis of distal metastasis in non-small cell lung cancer by low-dose CT based radiomics and deep learning signatures.

This study aimed to develop and validate radiomics and deep learning (DL) signatures for predicting distal metastasis (DM) of non-small cell lung cancer (NSCLC) in low-dose computed tomography (LDCT). Images and clinical data were retrospectively collected for 381 NSCLC patients and prospectively collected for 114 patients at the Fifth Affiliated Hospital of Sun Yat-Sen University. Additionally, we enrolled 179 patients from the Jiangmen Central Hospital to externally validate the signatures. Machine-learning algorithms were employed to develop radiomics signature while the DL signature was developed using neural architecture search. The diagnostic efficiency was primarily quantified with the area under receiver operating characteristic curve (AUC). We interpreted the reasoning process of the radiomics signature and DL signature by radiomics voxel mapping and attention weight tracking. A total of 674 patients with pathologically-confirmed NSCLC were included from two institutions, with 143 of them having DM. The radiomics signature achieved AUCs of 0.885, 0.854, and 0.733 in the internal validation, prospective validation, and external validation while those for DL signature were 0.893, 0.786, and 0.780. The proposed signatures achieved a promising performance in predicting the DM of NSCLC and outperformed the approaches proposed in previous studies. Interpretability analysis revealed that both radiomics and DL signatures could detect the variations among voxels inside tumors, which helped in identifying the DM of NSCLC. Our study demonstrates the potential of LDCT-based radiomics and DL signatures for predicting DM in NSCLC. These signatures could help improve lung cancer screening regarding further diagnostic tests and treatment strategies.

Validating Siamese embedded neural networks with identical representations for efficient model convergence

Deep learning, neural architecture search, reinforcement learning, and embedded learning use scaled data, hyperspace, reward, and similarity to produce efficient converging neural networks. However, all these state-of-the-art frameworks require training to evaluate whether the examples are identical. Therefore, we propose our Probabilistic Asymmetric Convergence Network with Validation and Transform-Learning (PACNVT) framework that learns with fewer data, reduces the hyperspace with our validation technique and modified tangent activation function, reinforces the learning with our transform learning algorithms, and ascertains similarity independent of spatial and mask for transformational consistency. Furthermore, our framework generates neural networks that yield state-of-the-art accuracies on MNIST, OMNIGLOT, and CIFAR datasets. Moreover, representing regression as a similarity problem unveils previously unseen residual fluid intelligence patterns, yielding 99.92% accuracy with as little as 20 pairs.

Empowering EEG motor imagery classification with deep transfer learning approach

AbstractThe brain‐computer interface (BCI) enables individuals with impairments to interact with the real world without relying on the neuromuscular pathway. BCI leverages artificial intelligence (AI) models for control. It can capture brain activity patterns associated with mental processes and convert them into commands for actuators. One promising application of BCI is in rehabilitation centres. When compared to traditional methods integrated with machine learning (ML) approaches that were initially explored in the past decade, there is a relatively limited number of studies utilizing deep learning (DL) approaches in BCI applications. A novel approach has been developed for the automatic recognition of motor imagery (MI) tasks. This article introduces the improved sparrow search algorithm with deep transfer learning‐based EEG motor imagery classification (ISSADTL‐EEGMIC) algorithm for BCIs. The presented ISSADTL‐EEGMIC algorithm primarily focuses on the automated classification of motor imagery tasks. To achieve this, the ISSADTL‐EEGMIC method first preprocesses EEG signals using the wavelet packet decomposition (WPD) technique. Additionally, it employs the neural architecture search network (NASNet) technique for feature extraction. Moreover, the ISSADTL‐EEGMIC model performs the classification process using the stacked sparse autoencoder (SSAE) model. Furthermore, hyperparameter optimization of the SSAE model is performed using the ISSA technique, which helps achieve maximum classification performance. The results from the simulation of the ISSADTL‐EEGMIC algorithm have been examined using two benchmark datasets. The experimental findings suggest that it outperforms recent approaches in terms of performance.

AutoGAN-DSP: Stabilizing GAN architecture search with deterministic score predictors

Generative Adversarial Network (GAN) has been widely used in many research areas of computer vision, anomaly detection, translation, optimal control, etc. However, in most cases, its network architectures have been hand-picked based on human experiences. To this end, neural Architecture Search (NAS) that automatically finds architectures have attracted much attention to automate the architecture search. In this work, we show that the NAS for Generative Adversarial Network (GAN), also denoted Generative Adversarial Neural Architecture Search (GANAS), often suffers from unstable search due to its innate randomness in the performance evaluation process. We address the stability issue by introducing deterministic score predictors and develop a unified framework to simultaneously conduct the architecture search and the predictor training. Further we develop a novel 2-phase architecture and parameter selection process to balance computational cost and architecture performance. Through extensive experiments, we demonstrate that our proposed AutoGAN-DSP outperforms other RL-based GANAS schemes as well as stabilizing the search performance. Our code and datasets are available on GitHub (https://github.com/APinCan/GAN_Architecture_Search_with_Predictors).

PsoResNet: An improved PSO-based residual network search algorithm

Neural Architecture Search (NAS) methods are widely employed to address the time-consuming and costly challenges associated with manual operation and design of deep convolutional neural networks (DCNNs). Nonetheless, prevailing methods still encounter several pressing obstacles, including limited network architecture design, excessively lengthy search periods, and insufficient utilization of the search space. In light of these concerns, this study proposes an optimization strategy for residual networks that leverages an enhanced Particle swarm optimization algorithm. Primarily, low-complexity residual architecture block is employed as the foundational unit for architecture exploration, facilitating a more diverse investigation into network architectures while minimizing parameters. Additionally, we employ a depth initialization strategy to confine the search space within a reasonable range, thereby mitigating unnecessary particle exploration. Lastly, we present a novel approach for computing particle differences and updating velocity mechanisms to enhance the exploration of updated trajectories. This method significantly contributes to the improved utilization of the search space and the augmentation of particle diversity. Moreover, we constructed a crime-dataset comprising 13 classes to assess the effectiveness of the proposed algorithm. Experimental results demonstrate that our algorithm can design lightweight networks with superior classification performance on both benchmark datasets and the crime-dataset.

Towards Full Forward On-Tiny-Device Learning: A Guided Search for a Randomly Initialized Neural Network

In the context of TinyML, many research efforts have been devoted to designing forward topologies to support On-Device Learning. Reaching this target would bring numerous advantages, including reductions in latency and computational complexity, stronger privacy, data safety and robustness to adversarial attacks, higher resilience against concept drift, etc. However, On-Device Learning on resource constrained devices poses severe limitations to computational power and memory. Therefore, deploying Neural Networks on tiny devices appears to be prohibitive, since their backpropagation-based training is too memory demanding for their embedded assets. Using Extreme Learning Machines based on Convolutional Neural Networks might be feasible and very convenient, especially for Feature Extraction tasks. However, it requires searching for a randomly initialized topology that achieves results as good as those achieved by the backpropagated model. This work proposes a novel approach for automatically composing an Extreme Convolutional Feature Extractor, based on Neural Architecture Search and Bayesian Optimization. It was applied to the CIFAR-10 and MNIST datasets for evaluation. Two search spaces have been defined, as well as a search strategy that has been tested with two surrogate models, Gaussian Process and Random Forest. A performance estimation strategy was defined, keeping the feature set computed by the MLCommons-Tiny benchmark ResNet as a reference model. In as few as 1200 search iterations, the proposed strategy was able to achieve a topology whose extracted features scored a mean square error equal to 0.64 compared to the reference set. Further improvements are required, with a target of at least one order of magnitude decrease in mean square error for improved classification accuracy. The code is made available via GitHub to allow for the reproducibility of the results reported in this paper.

RCF-ST: RICHER CONVOLUTIONAL FEATURES NETWORK WITH STRUCTURAL TUNING FOR THE EDGE DETECTION ON NATURAL IMAGES

Context. The problem of automating of the edge detection on natural images in intelligent systems is considered. The subject of the research is the deep learning convolutional neural networks for edge detection on natural images.
 Objective. The objective of the research is to improve the edge detection performance of natural images by structural tuning the richer convolutional features network architecture.
 Method. In general, the edge detection performance is influenced by a neural network architecture. To automate the design of the network structure in the paper a structural tuning of a neural network is applied. Computational costs of a structural tuning are incomparably less compared with neural architecture search, but a higher qualification of the researcher is required, and the resulting solution will be suboptimal. In this research it is successively applied first a destructive approach and then a constructive approach to structural tuning of the based architecture of the RCF neural network. The constructive approach starts with a simple architecture network. Hidden layers, nodes, and connections are added to expand the network. The destructive approach starts with a complex architecture network. Hidden layers, nodes, and connections are then deleted to contract the network. The structural tuning of the richer convolutional features network includes: (1) reducing the number of convolutional layers; (2) reducing the number of convolutions in convolutional layers; (3) removing at each stage the sigmoid activation function with subsequent calculation of the loss function; (4) addition of the batch normalization layers after convolutional layers; (5) including the ReLU activation functions after the added batch normalization layers. The obtained neural network is named RCF-ST. The initial color images were scaled to the specified size and then inputted in the neural network. The advisability of each of the proposed stages of network structural tuning was reseached by estimating the edge detection performance using the confusion matrix elements and Figure of Merit. The advisability of a structural tuning of the neural network as a whole was estimated by comparing it with methods known from the literature using the Optimal Dataset Scale and Optimal Image Scale.
 Results. The proposed convolutional neural network has been implemented in software and researched for solving the problem of edge detection on natural images. The structural tuning technique may be used for informed design of the neural network architectures for other artificial intelligence problems.
 Conclusions. The obtained RCF-ST network allows to improve the performance of edge detection on natural images. RCF-ST network is characterized by a significantly fewer parameters compared to the RCF network, which makes it possible to reduce the resource consumption of the network. Besides, RCF-ST network ensures the enhancing of the robustness of edge detection on texture background.

Evolutionary neural architecture search based on group whitening residual ConvNet and large kernel subspace attention module

Evolutionary Neural Architecture Search (ENAS) has achieved prominent achievements over the past years. Unfortunately, a majority of existing ENAS algorithms require enormous computational resources to design architectures. To achieve efficient search and superior performance, this paper proposes a novel attention mechanism and a compact architecture backbone for the evolutionary search of neural networks. A ‘Group Whitening Residual Block’ (GRBlock) is designed to utilize the merits of whitening operation and prevents the disadvantages from mini-batch normalization, thus enhancing the robustness of network architectures. In particular, a novel but effective ‘Large Kernel Subspace Attention Mechanism’ (LKSAM) is proposed to enhance the efficiency and representational capacity of neural networks, which is capable of extracting the complex interaction information between diverse channels with few parameters while achieves the multi-scale and multi-frequency feature representation. Experimental results on broadly used datasets reveal the superiority of our method in classification accuracy, architecture complexity, and computational overheads. Furthermore, the proposed algorithm is evaluated on traffic sign recognition task, and outperforms the manually-designed method and other algorithms.

An innovative fusion feature method of spectrum and visual image for diagnosing ‘Akizuki’ pear cork spot disorder

Cork spot disorder is a common physiological disease that severely affects the quality and yield of the ‘Akizuki’ pear. In this study, we propose a novel method, Near-infrared spectra and Visual images with Neural Architecture Search (NV-NAS), to achieve a highly accurate diagnosis of cork spot disorder. This method automatically generates fusion features by combining near-infrared spectral and visual image features and then searches for the best-performing model structure. The best result was obtained with near-infrared spectral features preprocessed by first derivatives (FD) and the visual image features extracted by DenseNet121 in the NV-NAS method. On the validation set, the accuracy was 95.5%, and on the test set, the accuracy was 88.2%. Compared to the modeling results of a single feature, the accuracy improved by 26.3% over spectra and by 11.2% over images on the test set. Additionally, in experiments with models of different depths, deep image features were more conducive to feature fusion, with accuracy improvements of 21.7% and 30.1% compared to features from the middle or bottom layers, respectively. The results indicate that the model built on the fusion feature shows high generalization performance and holds significant reference value for the more accurate diagnosis of the ‘Akizuki’ pear cork spot disorder.

A Gradient-Guided Evolutionary Neural Architecture Search.

Neural architecture search (NAS) is a popular method that can automatically design deep neural network structures. However, designing a neural network using NAS is computationally expensive. This article proposes a gradient-guided evolutionary NAS (GENAS) to design convolutional neural networks (CNNs) for image classification. GENAS is a hybrid algorithm that combines evolutionary global and local search operators to evolve a population of subnets sampled from a supernet. Each candidate architecture is encoded as a table describing which operations are associated with the edges between nodes signifying feature maps. Besides, evolutionary optimization uses novel crossover and mutation operators to manipulate the subnets using the proposed tabular encoding. Every n generations, the candidate architectures undergo a local search inspired by differentiable NAS. GENAS is designed to overcome the limitations of both evolutionary and gradient descent NAS. This algorithmic structure enables the performance assessment of the candidate architecture without retraining, thus limiting the NAS calculation time. Furthermore, subnet individuals are decoupled during evaluation to prevent strong coupling of operations in the supernet. The experimental results indicate that the searched structures achieve test errors of 2.45%, 16.86%, and 23.9% on CIFAR-10/100/ImageNet datasets and it costs only 0.26 GPU days on a graphic card. GENAS can effectively expedite the training and evaluation processes and obtain high-performance network structures.

Deeply Supervised Block-Wise Neural Architecture Search.

Neural architecture search (NAS) has shown great promise in automatically designing neural network models. Recently, block-wise NAS has been proposed to alleviate deep coupling problem between architectures and weights existed in the well-known weight-sharing NAS, by training the huge weight-sharing supernet block-wisely. However, the existing block-wise NAS methods, which resort to either supervised distillation or self-supervised contrastive learning scheme to enable block-wise optimization, take massive computational cost. To be specific, the former introduces an external high-capacity teacher model, while the latter involves supernet-scale momentum model and requires a long training schedule. Considering this, in this work, we propose a resource-friendly deeply supervised block-wise NAS (DBNAS) method. In the proposed DBNAS, we construct a lightweight deeply-supervised module after each block to enable a simple supervised learning scheme and leverage ground-truth labels to indirectly supervise optimization of each block progressively. Besides, the deeply-supervised module is specifically designed as structural and functional condensation of the supernet, which establishes global awareness for progressive block-wise optimization and helps search for promising architectures. Experimental results show that the DBNAS method only takes less than 1 GPU day to search out promising architectures on the ImageNet dataset with less GPU memory footprint than the other block-wise NAS works. The best-performing model among the searched DBNAS family achieves 75.6% Top-1 accuracy on ImageNet, which is competitive with the state-of-the-art NAS models. Moreover, our DBNAS family models also achieve good transfer performance on CIFAR-10/100, as well as two downstream tasks: object detection and semantic segmentation.

Zero-Shot Neural Architecture Search: Challenges, Solutions, and Opportunities.

Recently, zero-shot (or training-free) Neural Architecture Search (NAS) approaches have been proposed to liberate NAS from the expensive training process. The key idea behind zero-shot NAS approaches is to design proxies that can predict the accuracy of some given networks without training the network parameters. The proxies proposed so far are usually inspired by recent progress in theoretical understanding of deep learning and have shown great potential on several datasets and NAS benchmarks. This paper aims to comprehensively review and compare the state-of-the-art (SOTA) zero-shot NAS approaches, with an emphasis on their hardware awareness. To this end, we first review the mainstream zero-shot proxies and discuss their theoretical underpinnings. We then compare these zero-shot proxies through large-scale experiments and demonstrate their effectiveness in both hardware-aware and hardware-oblivious NAS scenarios. Finally, we point out several promising ideas to design better proxies. Our source code and the list of related papers are available on https://github.com/SLDGroup/survey-zero-shot-nas.

Generalizable Reconstruction for Accelerating MR Imaging via Federated Learning with Neural Architecture Search.

Heterogeneous data captured by different scanning devices and imaging protocols can affect the generalization performance of the deep learning magnetic resonance (MR) reconstruction model. While a centralized training model is effective in mitigating this problem, it raises concerns about privacy protection. Federated learning is a distributed training paradigm that can utilize multi-institutional data for collaborative training without sharing data. However, existing federated learning MR image reconstruction methods rely on models designed manually by experts, which are complex and computationally expensive, suffering from performance degradation when facing heterogeneous data distributions. In addition, these methods give inadequate consideration to fairness issues, namely ensuring that the model's training does not introduce bias towards any specific dataset's distribution. To this end, this paper proposes a generalizable federated neural architecture search framework for accelerating MR imaging (GAutoMRI). Specifically, automatic neural architecture search is investigated for effective and efficient neural network representation learning of MR images from different centers. Furthermore, we design a fairness adjustment approach that can enable the model to learn features fairly from inconsistent distributions of different devices and centers, and thus facilitate the model to generalize well to the unseen center. Extensive experiments show that our proposed GAutoMRI has better performances and generalization ability compared with seven state-of-the-art federated learning methods. Moreover, the GAutoMRI model is significantly more lightweight, making it an efficient choice for MR image reconstruction tasks. The code will be made available at https://github.com/ternencewu123/GAutoMRI.

CLASH: Complementary Learning with Neural Architecture Search for Gait Recognition.

Gait recognition, which aims at identifying individuals by their walking patterns, has achieved great success based on silhouette. The binary silhouette sequence encodes the walking pattern within the sparse boundary representation. Therefore, most pixels in the silhouette are under-sensitive to the walking pattern since the sparse boundary lacks dense spatial-temporal information, which is suitable to be represented with dense texture. To enhance the sensitivity to the walking pattern while maintaining the robustness of recognition, we present a Complementary Learning with neural Architecture SearcH (CLASH) framework, consisting of walking pattern sensitive gait descriptor named dense spatial-temporal field (DSTF) and neural architecture search based complementary learning (NCL). Specifically, DSTF transforms the representation from the sparse binary boundary into the dense distance-based texture, which is sensitive to the walking pattern at the pixel level. Further, NCL presents a task-specific search space for complementary learning, which mutually complements the sensitivity of DSTF and the robustness of the silhouette to represent the walking pattern effectively. Extensive experiments demonstrate the effectiveness of the proposed methods under both in-the-lab and in-the-wild scenarios. On CASIA-B, we achieve rank-1 accuracy of 98.8%, 96.5%, and 89.3% under three conditions. On OU-MVLP, we achieve rank-1 accuracy of 91.9%. Under the latest in-the-wild datasets, we outperform the latest silhouette-based methods by 16.3% and 19.7% on Gait3D and GREW, respectively.

Similarity surrogate-assisted evolutionary neural architecture search with dual encoding strategy

<abstract><p>Neural architecture search (NAS), a promising method for automated neural architecture design, is often hampered by its overwhelming computational burden, especially the architecture evaluation process in evolutionary neural architecture search (ENAS). Although there are surrogate models based on regression or ranking to assist or replace the neural architecture evaluation process in ENAS to reduce the computational cost, these surrogate models are still affected by poor architectures and are not able to accurately find good architectures in a search space. To solve the above problems, we propose a novel surrogate-assisted NAS approach, which we call the similarity surrogate-assisted ENAS with dual encoding strategy (SSENAS). We propose a surrogate model based on similarity measurement to select excellent neural architectures from a large number of candidate architectures in a search space. Furthermore, we propose a dual encoding strategy for architecture generation and surrogate evaluation in ENAS to improve the exploration of well-performing neural architectures in a search space and realize sufficiently informative representations of neural architectures, respectively. We have performed experiments on NAS benchmarks to verify the effectiveness of the proposed algorithm. The experimental results show that SSENAS can accurately find the best neural architecture in the NAS-Bench-201 search space after only 400 queries of the tabular benchmark. In the NAS-Bench-101 search space, it can also get results that are comparable to other algorithms. In addition, we conducted a large number of experiments and analyses on the proposed algorithm, showing that the surrogate model measured via similarity can gradually search for excellent neural architectures in a search space.</p></abstract>

A Fast Evolutionary Knowledge Transfer Search for Multiscale Deep Neural Architecture.

The emergence of neural architecture search (NAS) algorithms has removed the constraints on manually designed neural network architectures, so that neural network development no longer requires extensive professional knowledge, trial and error. However, the extremely high computational cost limits the development of NAS algorithms. In this article, in order to reduce computational costs and to improve the efficiency and effectiveness of evolutionary NAS (ENAS) is investigated. In this article, we present a fast ENAS framework for multiscale convolutional networks based on evolutionary knowledge transfer search (EKTS). This framework is novel, in that it combines global optimization methods with local optimization methods for search, and searches a multiscale network architecture. In this article, evolutionary computation is used as a global optimization algorithm with high robustness and wide applicability for searching neural architectures. At the same time, for fast search, we combine knowledge transfer and local fast learning to improve the search speed. In addition, we explore a multiscale gray-box structure. This gray box structure combines the Bandelet transform with convolution to improve network approximation, learning, and generalization. Finally, we compare the architectures with more than 40 different neural architectures, and the results confirmed its effectiveness.