Architecture Search Research Articles

Momentum recursive DARTS

DARTS has emerged as a popular method for neural architecture search (NAS) owing to its efficiency and simplicity. It employs gradient-based bi-level optimization to iteratively optimize the upper-level architecture parameters and lower-level super-network weights. The key challenge in DARTS is the accurate estimation of gradients for two-level object functions, leading to significant errors in gradient approximation. To address this issue, we propose a new approach, MR-DARTS, that incorporates a momentum term and a recursive scheme to improve gradient estimation. Specifically, we leverage historical information by using a running average of past observed gradients to enhance the quality of current gradient estimation in both upper-level and lower-level functions. Our theoretical analysis shows that the variance of our estimated gradient decreases with each iteration. By utilizing momentum and a recursive scheme, MR-DARTS effectively controls the error in stochastic gradient updates that result from inaccurate gradient estimation. Furthermore, we utilize the Neumann series approximation and Hessian Vector Product scheme to reduce computational requirements and memory usage. We evaluate our proposed method on several benchmarks and demonstrate its effectiveness through comprehensive experiments.

Neural network architecture search model for thermal radiation in dense particulate systems

In the CFD-DEM simulation of the many dense particulate systems, particle-scale thermal radiation is an important heat transfer mode under high temperatures. In this work, neural network architecture search model with different layer connection matrix is applied to the view factor regression of the thermal radiation in dense particulate systems. Deep neural networks with 3346 feasible architectures are evaluated by the HyperBand algorithm to find the local optimal solution. Neural architecture search model trained by the big data of the view factor gives a good prediction of the macroscopic radiative properties and it is in general agreement with the empirical correlations and experimental data. The particle–wall radiation decreases strongly with the distance and the maximum interaction depth is about 2.0 times the sphere diameter. The trained deep neural network model provides an efficient data-driven closure to discuss the thermal radiation of the particle–particle and particle–wall interactions in particle bed.

Enhanced Gradient for Differentiable Architecture Search.

In recent years, neural architecture search (NAS) methods have been proposed for the automatic generation of task-oriented network architecture in image classification. However, the architectures obtained by existing NAS approaches are optimized only for classification performance and do not adapt to devices with limited computational resources. To address this challenge, we propose a neural network architecture search algorithm aiming to simultaneously improve the network performance and reduce the network complexity. The proposed framework automatically builds the network architecture at two stages: block-level search and network-level search. At the stage of block-level search, a gradient-based relaxation method is proposed, using an enhanced gradient to design high-performance and low-complexity blocks. At the stage of network-level search, an evolutionary multiobjective algorithm is utilized to complete the automatic design from blocks to the target network. The experimental results demonstrate that our method outperforms all evaluated hand-crafted networks in image classification, with an error rate of 3.18% on Canadian Institute for Advanced Research (CIFAR10) and an error rate of 19.16% on CIFAR100, both at network parameter size less than 1 M. Obviously, compared with other NAS methods, our method offers a tremendous reduction in designed network architecture parameters.

Wheat Leaf Disease Detection: A Lightweight Approach with Shallow CNN Based Feature Refinement

Improving agricultural productivity is essential due to rapid population growth, making early detection of crop diseases crucial. Although deep learning shows promise in smart agriculture, practical applications for identifying wheat diseases in complex backgrounds are limited. In this paper, we propose CropNet, a hybrid method that utilizes Red, Green, and Blue (RGB) imaging and a transfer learning approach combined with shallow convolutional neural networks (CNN) for further feature refinement. To develop our customized model, we conducted an extensive search for the optimal deep learning architecture. Our approach involves freezing the pre-trained model for feature extraction and adding a custom trainable CNN layer. Unlike traditional transfer learning, which typically uses trainable dense layers, our method integrates a trainable CNN, deepening the architecture. We argue that pre-trained features in transfer learning are better suited for a custom shallow CNN followed by a fully connected layer, rather than being fed directly into fully connected layers. We tested various architectures for pre-trained models including EfficientNetB0 and B2, DenseNet, ResNet50, MobileNetV2, MobileNetV3-Small, and Inceptionv3. Our approach combines the strengths of pre-trained models with the flexibility of custom architecture design, offering efficiency, effective feature extraction, customization options, reduced overfitting, and differential learning rates. It distinguishes itself from classical transfer learning techniques, which typically fine-tune the entire pre-trained network. Our aim is to provide a lightweight model suitable for resource-constrained environments, capable of delivering outstanding results. CropNet achieved 99.80% accuracy in wheat disease detection with reduced training time and computational cost. This efficient performance makes CropNet promising for practical implementation in resource-constrained agricultural settings, benefiting farmers and enhancing production.

Explainable coronary artery disease prediction model based on AutoGluon from AutoML framework.

This study focuses on the innovative application of Automated Machine Learning (AutoML) technology in cardiovascular medicine to construct an explainable Coronary Artery Disease (CAD) prediction model to support the clinical diagnosis of CAD. This study utilizes a combined data set of five public data sets related to CAD. An ensemble model is constructed using the AutoML open-source framework AutoGluon to evaluate the feasibility of AutoML in constructing a disease prediction model in cardiovascular medicine. The performance of the ensemble model is compared against individual baseline models. Finally, the disease prediction ensemble model is explained using SHapley Additive exPlanations (SHAP). The experimental results show that the AutoGluon-based ensemble model performs better than the individual baseline models in predicting CAD. It achieved an accuracy of 0.9167 and an AUC of 0.9562 in 4-fold cross-bagging. SHAP measures the importance of each feature to the prediction of the model and explains the prediction results of the model. This study demonstrates the feasibility and efficacy of AutoML technology in cardiovascular medicine and highlights its potential in disease prediction. AutoML reduces the barriers to model building and significantly improves prediction accuracy. Additionally, the integration of SHAP enhances model transparency and explainability, which is critical to ensuring model credibility and widespread adoption in cardiovascular medicine.

Latency-Aware Neural Architecture Performance Predictor With Query-to-Tier Technique

Neural Architecture Search (NAS) is a powerful tool for automating effective image and video processing DNN designing. The ranking of the accuracy has been advocated to design an efficient performance predictor for NAS. The previous contrastive method solves the ranking problem by comparing pairs of architectures and predicting their relative performance. However, it only focuses on the rankings between the two involved architectures and neglects the overall quality distributions of the search space, which may suffer generalization issues. On the contrary, we propose to let the performance predictor concentrate on the global quality level of specific architecture, and learn the tier embeddings of the whole search space automatically with learnable queries. The proposed method, dubbed as Neural Architecture Ranker with Query-to-Tier technique (NARQ2T), explores the quality tiers of the search space globally and classifies each individual to the tier they belong to. Thus, the predictor gains knowledge of the performance distributions of the search space which helps to generalize its ranking ability to the datasets more easily. Thanks to the encoder-decoder design, our method is able to predict the latency of the searched model without deteriorating the performance prediction. Meanwhile, the global quality distribution facilitates the search phase by directly sampling candidates according to the statistics of quality tiers, which is free of training a search algorithm, e.g., Reinforcement Learning or Evolutionary Algorithm, thus it simplifies the NAS pipeline and saves the computational overheads. The proposed NARQ2T achieves state-of-the-art performance on two widely used datasets for NAS research. Moreover, extensive experiments have validated the efficacy of the designed method.

Lightweight Deep Learning: An Overview

With the recent success of the deep neural networks (DNNs) in the field of artificial intelligence, the urge of deploying DNNs has drawn tremendous attention because it can benefit a wide range of applications on edge or embedded devices. Lightweight deep learning (DL) indicates the procedures of compressing DNN models into more compact ones, which are suitable to be executed on edge devices due to their limited resources and computational capabilities while maintaining comparable performance as the original. Currently, the approaches of model compression include but not limited to network pruning, quantization, knowledge distillation, neural architecture search. In this work, we present a fresh overview to summarize recent development and challenges for model compression.

Computer vision for eye diseases detection using pre‐trained deep learning techniques and raspberry Pi

AbstractEarly diagnosis of eye diseases is very important to prevent visual impairment and guide appropriate treatment methods. This paper presents a unique approach that can detect numerous eye diseases automatically. Initially, this approach used the pre‐trained ImageNet models that provides various pre‐trained models for training the acquired data. The existing data sets are composed of 645 data images acquired clinically, represented by two groups of subjects as healthy and others holding the proposed eye defect like cataracts, foreign bodies, glaucoma, subconjunctival haemorrhage, and viral conjunctivitis. Followed by comparisons of the pre‐trained model's coefficients and prediction performance. Later, the first‐class execution model is integrated within the Raspberry Pi staging and the real‐time digital camera detection. The evaluation process used the confusion matrix, model accuracy, precision factor, recall coefficient, F1 score, and the Matthews Correlation Coefficient (MCC). Resulting in the performance of these pre‐trained ImageNet models used in this study represented by 93% (InceptionResNetV2), 90% (MobileNet), 86% (Residual Network ResNet50), 85% (InceptionV3), 78% (Visual Geometry Group VGG19), and 72% (Neural Architecture Search Network NASNetMobile). The results show that the InceptionResNetV2 achieved the highest performance. This proposed approach shows its efficiency and strength by early detection of the subject's unhealthy eyes through real‐time monitoring in the field of ophthalmology.

Carbon-Efficient Neural Architecture Search

This work presents a novel approach to neural architecture search (NAS) that aims to reduce energy costs and increase carbon efficiency during the model design process. The proposed framework, called carbon-efficient NAS (CE-NAS), consists of NAS evaluation algorithms with different energy requirements, a multi-objective optimizer, and a heuristic GPU allocation strategy. CE-NAS dynamically balances energy-efficient sampling and energy-consuming evaluation tasks based on current carbon emissions. Using a recent NAS benchmark dataset and two carbon traces, our trace-driven simulations demonstrate that CE-NAS achieves better carbon and search efficiency than the three baselines.

Fast Filter Pruning via Coarse-to-Fine Neural Architecture Search and Contrastive Knowledge Transfer.

Filter pruning is the most representative technique for lightweighting convolutional neural networks (CNNs). In general, filter pruning consists of the pruning and fine-tuning phases, and both still require a considerable computational cost. So, to increase the usability of CNNs, filter pruning itself needs to be lightweighted. For this purpose, we propose a coarse-to-fine neural architecture search (NAS) algorithm and a fine-tuning structure based on contrastive knowledge transfer (CKT). First, candidates of subnetworks are coarsely searched by a filter importance scoring (FIS) technique, and then the best subnetwork is obtained by a fine search based on NAS-based pruning. The proposed pruning algorithm does not require a supernet and adopts a computationally efficient search process, so it can create a pruned network with higher performance at a lower cost than the existing NAS-based search algorithms. Next, a memory bank is configured to store the information of interim subnetworks, i.e., by-products of the above-mentioned subnetwork search phase. Finally, the fine-tuning phase delivers the information of the memory bank through a CKT algorithm. Thanks to the proposed fine-tuning algorithm, the pruned network accomplishes high performance and fast convergence speed because it can take clear guidance from the memory bank. Experiments on various datasets and models prove that the proposed method has a significant speed efficiency with reasonable performance leakage over the state-of-the-art (SOTA) models. For example, the proposed method pruned the ResNet-50 trained on Imagenet-2012 up to 40.01% with no accuracy loss. Also, since the computational cost amounts to only 210 GPU hours, the proposed method is computationally more efficient than SOTA techniques. The source code is publicly available at https://github.com/sseung0703/FFP.

IoT Forensic Cyber Activities Detection and Prevention with Automated Machine Learning Model

The Internet of Things (IoT) has been deployed in a vast range of applications with exponential increases in data size and complexity. Existing forensic techniques are not effective for the accuracy and detection rate of security issues in IoT forensics. Cyber forensic comprises huge volume constraints that are processing huge volumes of data in the Information and Communication Technology (ICT) comprised of IoT devices and platforms. Trust blockchain is effective technology those are utilized to assess the tamper-proof records in all transaction in the IoT environment. With the implementation of trust blockchain the record and transaction are processed with a distributed ledger that is managed by the network nodes. The challenge associated with the trust blockchain in IoT forensics is cost and security. To achieve significant cost-effectiveness organizations, need to evaluate the risks and benefits associated with IoT forensics in the trust blockchain technology. In this paper, developed a Block Chain Enabled Cyber-Physical system with distributed storage. The developed Blockchain model is termed as Integrated Hadoop Blockchain Forensic Machin Learning (IHBF-ML). The IHBF-ML model uses the Hadoop Distributed File System (HDFS) with cyberspace to improve security. Within the IHBF-ML model, IoT data communication is established with the smart contract. The smart contract-based blockchain process uses the Machine Learning model integrated with Cat Boost classification model for anomaly detection. Cost in IoT forensic is minimized with the parallel processing of the data through MapReduce Framework for the traffic translation, extraction, and analysis of the dynamic feature traffic from the IoT environment. The experimental analysis stated that constructed IHBF-ML model reduces the cost by ~25% than the other conventional blockchain Ethereum and EOS.

Comparing machine learning screening approaches using clinical data and cytokine profiles for COVID-19 in resource-limited and resource-abundant settings

Accurate screening of COVID-19 infection status for symptomatic patients is a critical public health task. Although molecular and antigen tests now exist for COVID-19, in resource-limited settings, screening tests are often not available. Furthermore, during the early stages of the pandemic tests were not available in any capacity. We utilized an automated machine learning (ML) approach to train and evaluate thousands of models on a clinical dataset consisting of commonly available clinical and laboratory data, along with cytokine profiles for patients (n = 150). These models were then further tested for generalizability on an out-of-sample secondary dataset (n = 120). We were able to develop a ML model for rapid and reliable screening of patients as COVID-19 positive or negative using three approaches: commonly available clinical and laboratory data, a cytokine profile, and a combination of the common data and cytokine profile. Of the tens of thousands of models automatically tested for the three approaches, all three approaches demonstrated > 92% sensitivity and > 88 specificity while our highest performing model achieved 95.6% sensitivity and 98.1% specificity. These models represent a potential effective deployable solution for COVID-19 status classification for symptomatic patients in resource-limited settings and provide proof-of-concept for rapid development of screening tools for novel emerging infectious diseases.

AFTLNet: An efficient adaptive forgery traces learning network for deep image inpainting localization

Deep-learning-based image inpainting repairs a region with visually believable content, leaving behind imperceptible traces. Since deep image inpainting approaches can malevolently remove key objects and erase visible copyright watermarks, the desire for an effective method to distinguish the inpainted regions has become urgent. In this work, we propose an adaptive forgery trace learning network (AFTLN), which consists of two subblocks: the adaptive block and the Densenet block. Specifically, the adaptive block exploits an adaptive difference convolution to maximize the forgery traces by iteratively updating its weights. Meanwhile, the Densenet block improves the feature weights and reduces the impact of noise on the forgery traces. An image-inpainting detector, namely AFTLNet, is designed by integrating AFTLN with neural architecture search, and global and local attention modules, which aims to find potential tampered regions, enhance feature consistency, and reduce intra-class differences, respectively. The experimental results present that our proposed AFTLNet exceeds existing inpainting detection approaches. Finally, an inpainting dataset of 26K image pairs is constructed for future research. The dataset is available at https://pan.baidu.com/s/10SRJeQBNnTHJXvxl8xzHcg with password: 1234.

Component importance preference-based evolutionary graph neural architecture search

Recently, graph neural architecture search (GNAS) has become an increasingly hot research topic as a promising technique for automatically searching graph neural networks (GNNs) with no or little domain expertise. The search space and optimization strategy are the core factors of GNAS. However, the search space of existing GNAS methods is limited, and their optimization strategies treat components indiscriminately. In the current study, a more expressive search space is first designed. Subsequently, it is illustrated that components contribute different importance to data-specific tasks or datasets, and this study assumes component importance as a probability parameter. To this end, a component importance preference-based evolutionary GNAS method (called CIPE) is proposed. CIPE defines component importance and its preference selection and updating method. Subsequently, the designed importance preference-guided multipoint crossover and multistrategy mutation operators are applied to the evolutionary process. Finally, the effectiveness of CIPE is verified for transductive and inductive tasks. The experimental results demonstrate the validity of the component importance assumption and the superiority of CIPE compared with the state-of-the-art handcrafted GNNs and GNAS methods on all eight datasets. The mean accuracy obtained by CIPE on the datasets Cora, CiteSeer, PubMed, Cornell, Texas, Wisconsin, and Chameleon is 83.84%, 73.23%, 80.28%, 78.38%, 86.49%, 82.35%, and 73.59%, respectively. Specifically, the mean accuracy is improved by 2.71% and 5.28% on the datasets Texas and Chameleon, respectively. The mean F1-score obtained by CIPE on the dataset PPI is 99.37%, with an improvement of 0.24%. The code is available at https://github.com/chnyliu/CIPE.

GLNAS: Greedy Layer-wise Network Architecture Search for low cost and fast network generation

The process of applying machine learning algorithms to practical problems can be a challenging and tedious task for non-experts. Previous research has sought to alleviate this burden by introducing automated machine learning techniques, including Network Architecture Search (NAS) and Differentiable Architecture Search (DARTS). However, these methods use a fixed number of layers and predefined skip connections which impose limitations on the generation of an optimal network architecture. In this paper, we propose a novel approach called Greedy Layer-wise Network Architecture Search (GLNAS), which trains network layers one after another and evaluates the network’s performance after each layer is added. GLNAS also assesses the effectiveness of skip connections between layers by testing various outputs of previous layers as an input to the current layer. Our experiment results demonstrate that the network generated by GLNAS requires fewer parameters (i.e., 3.5 millions in both CIFAR-10 and CIFAR-100 datasets) and GPU resources during the searching phase (i.e., 0.17 and 0.24 GPU days in CIFAR-10 and CIFAR-100 datasets respectively) than many existing methods.

Multiscale Feature Search-Based Graph Convolutional Network for Hyperspectral Image Classification

With the development of hyperspectral sensors, the availability of hyperspectral images (HSIs) has increased significantly, prompting advancements in deep learning-based hyperspectral image classification (HSIC) methods. Recently, graph convolutional networks (GCNs) have been proposed to process graph-structured data in non-Euclidean domains, and have been used for HSIC. The superpixel segmentation should be implemented first in the GCN-based methods, however, it is difficult to manually select the optimal superpixel segmentation sizes to obtain the useful information for classification. To solve this problem, we constructed a HSIC model based on a multiscale feature search-based graph convolutional network (MFSGCN) in this study. Firstly, pixel-level features of HSIs are extracted sequentially using 3D asymmetric decomposition convolution and 2D convolution. Then, superpixel-level features at different scales are extracted using multilayer GCNs. Finally, the neural architecture search (NAS) method is used to automatically assign different weights to different scales of superpixel features. Thus, a more discriminative feature map is obtained for classification. Compared with other GCN-based networks, the MFSGCN network can automatically capture features and obtain higher classification accuracy. The proposed MFSGCN model was implemented on three commonly used HSI datasets and compared to some state-of-the-art methods. The results confirm that MFSGCN effectively improves accuracy.

Gas identification using electronic nose via gramian-angular-field-based image conversion and convolutional neural networks architecture search

Recent years have witnessed the splendid performance of deep learning methods used in gas recognition for electronic noses (E-nose). In addition to effective feature extraction, the architecture of the deep neural network plays a vital role. Currently, most applied network structures are hand-crafted by human experts, which is time-consuming and problem-dependent, making it necessary to design the structures of neural networks according to specific demands. In this work, a genetic algorithm with particle swarm optimization (GA-PSO), which possesses promising optimization capabilities, is applied to search for effective deep convolutional neural networks (CNNs) for gas classification based on E-nose technology. A novel image transformation strategy using Gramian angular field and a hybrid cost-saving method is employed in the search process, enabling adaptive and efficient CNN search on gas datasets. With the proposed methods, we can achieve an average classification accuracy of over 90 % on two public gas datasets, while also significantly reducing the model size compared to state-of-the-art CNNs. By using these novel strategies, our approach surpasses random search and basic PSO algorithm in achieving the global optimal solution, higher and more stable accuracy, and faster convergence in pattern recognition using E-nose. Our work suggests that the proposed method can quickly identify excellent CNN structures for E-nose applications.

AutoAMS: Automated attention-based multi-modal graph learning architecture search

Multi-modal attention mechanisms have been successfully used in multi-modal graph learning for various tasks. However, existing attention-based multi-modal graph learning (AMGL) architectures heavily rely on manual design, requiring huge effort and expert experience. Meanwhile, graph neural architecture search (GNAS) has made great progress toward automatically designing graph-based learning architectures. However, it is challenging to directly adopt existing GNAS methods to search for better AMGL architectures because of the search spaces that only focus on designing graph neural network architectures and the search objective that ignores multi-modal interactive information between modalities and long-term content dependencies within different modalities. To address these issues, we propose an automated attention-based multi-modal graph learning architecture search (AutoAMS) framework, which can automatically design the optimal AMGL architectures for different multi-modal tasks. Specifically, we design an effective attention-based multi-modal (AM) search space consisting of four sub-spaces, which can jointly support the automatic search of multi-modal attention representation and other components of multi-modal graph learning architecture. In addition, a novel search objective based on an unsupervised multi-modal reconstruction loss and task-specific loss is introduced to search and train AMGL architectures. The search objective can extract the global features and capture multi-modal interactions from multiple modalities. The experimental results on multi-modal tasks show strong evidence that AutoAMS is capable of designing high-performance AMGL architectures.

Adaptive multi-scale Graph Neural Architecture Search framework

Graph neural networks (GNNs) have gained significant attention for their ability to learn representations from graph-structured data, in which message passing and feature fusion strategies play an essential role. However, traditional Graph Neural Architecture Search (GNAS) mainly focuses on optimization with a static perceptive field to ease the search process. To efficiently utilize latent relationships between non-adjacent nodes as well as edge features, this work proposes a novel two-stage approach that is able to optimize GNN structures more effectively by adaptively aggregating neighborhood features in multiple scales. This adaptive multi-scale GNAS is able to assign optimal weights for different neighbors in different graphs and learning tasks. In addition, it takes latent relationships and edge features into message passing into account, and can incorporate different feature fusion strategies. Compared with traditional ones, our proposed approach can explore a much larger and more diversified search space efficiently. We also prove that traditional multi-hop GNNs are low-pass filters, which can lead to the removal of important low-frequency components of signals from remote neighbors in a graph, and they are not even expressive enough to distinguish some simple regular graphs, justifying the superiority of our approach. Experiments with seven datasets across three graph learning tasks, including graph regression, node classification, and graph classification, demonstrate that our method yields significant improvement compared with state-of-the-art GNAS approaches and human-designed GNN approaches. Specifically, for example, with our framework, the MAE of the 12-layer AM-GNAS was 0.102 for the ZINC dataset, yielding over 25% improvement.

Reinforcement Learning and Genetic Algorithm-Based Network Module for Camera-LiDAR Detection

Cameras and LiDAR sensors have been used in sensor fusion for robust object detection in autonomous driving. Object detection networks for autonomous driving are often trained again by adding or changing datasets aimed at robust performance. Repeat training is necessary to develop an efficient network module. Existing efficient network module development changes to hand design and requires much module design experience. For this, a neural architecture search was designed, but it takes much time and requires optimizing the design process. To solve this problem, we propose a two-stage optimization method for the offspring generation process in a neural architecture search based on reinforcement learning. In addition, we propose utilizing two split datasets to solve the fast convergence problem as the objective function of the genetic algorithm: source data (daytime, sunny) and target data (day/night, adversary weather). The proposed method is an efficient module generation method requiring less time than the NSGA-NET. We confirmed the performance improvement and the convergence speed reduction using the Dense dataset. Through experiments, it was proven that the proposed method generated an efficient module.