Architecture Search Research Articles

Component importance preference-based evolutionary graph neural architecture search

Recently, graph neural architecture search (GNAS) has become an increasingly hot research topic as a promising technique for automatically searching graph neural networks (GNNs) with no or little domain expertise. The search space and optimization strategy are the core factors of GNAS. However, the search space of existing GNAS methods is limited, and their optimization strategies treat components indiscriminately. In the current study, a more expressive search space is first designed. Subsequently, it is illustrated that components contribute different importance to data-specific tasks or datasets, and this study assumes component importance as a probability parameter. To this end, a component importance preference-based evolutionary GNAS method (called CIPE) is proposed. CIPE defines component importance and its preference selection and updating method. Subsequently, the designed importance preference-guided multipoint crossover and multistrategy mutation operators are applied to the evolutionary process. Finally, the effectiveness of CIPE is verified for transductive and inductive tasks. The experimental results demonstrate the validity of the component importance assumption and the superiority of CIPE compared with the state-of-the-art handcrafted GNNs and GNAS methods on all eight datasets. The mean accuracy obtained by CIPE on the datasets Cora, CiteSeer, PubMed, Cornell, Texas, Wisconsin, and Chameleon is 83.84%, 73.23%, 80.28%, 78.38%, 86.49%, 82.35%, and 73.59%, respectively. Specifically, the mean accuracy is improved by 2.71% and 5.28% on the datasets Texas and Chameleon, respectively. The mean F1-score obtained by CIPE on the dataset PPI is 99.37%, with an improvement of 0.24%. The code is available at https://github.com/chnyliu/CIPE.

GLNAS: Greedy Layer-wise Network Architecture Search for low cost and fast network generation

The process of applying machine learning algorithms to practical problems can be a challenging and tedious task for non-experts. Previous research has sought to alleviate this burden by introducing automated machine learning techniques, including Network Architecture Search (NAS) and Differentiable Architecture Search (DARTS). However, these methods use a fixed number of layers and predefined skip connections which impose limitations on the generation of an optimal network architecture. In this paper, we propose a novel approach called Greedy Layer-wise Network Architecture Search (GLNAS), which trains network layers one after another and evaluates the network’s performance after each layer is added. GLNAS also assesses the effectiveness of skip connections between layers by testing various outputs of previous layers as an input to the current layer. Our experiment results demonstrate that the network generated by GLNAS requires fewer parameters (i.e., 3.5 millions in both CIFAR-10 and CIFAR-100 datasets) and GPU resources during the searching phase (i.e., 0.17 and 0.24 GPU days in CIFAR-10 and CIFAR-100 datasets respectively) than many existing methods.

Multiscale Feature Search-Based Graph Convolutional Network for Hyperspectral Image Classification

With the development of hyperspectral sensors, the availability of hyperspectral images (HSIs) has increased significantly, prompting advancements in deep learning-based hyperspectral image classification (HSIC) methods. Recently, graph convolutional networks (GCNs) have been proposed to process graph-structured data in non-Euclidean domains, and have been used for HSIC. The superpixel segmentation should be implemented first in the GCN-based methods, however, it is difficult to manually select the optimal superpixel segmentation sizes to obtain the useful information for classification. To solve this problem, we constructed a HSIC model based on a multiscale feature search-based graph convolutional network (MFSGCN) in this study. Firstly, pixel-level features of HSIs are extracted sequentially using 3D asymmetric decomposition convolution and 2D convolution. Then, superpixel-level features at different scales are extracted using multilayer GCNs. Finally, the neural architecture search (NAS) method is used to automatically assign different weights to different scales of superpixel features. Thus, a more discriminative feature map is obtained for classification. Compared with other GCN-based networks, the MFSGCN network can automatically capture features and obtain higher classification accuracy. The proposed MFSGCN model was implemented on three commonly used HSI datasets and compared to some state-of-the-art methods. The results confirm that MFSGCN effectively improves accuracy.

Gas identification using electronic nose via gramian-angular-field-based image conversion and convolutional neural networks architecture search

Recent years have witnessed the splendid performance of deep learning methods used in gas recognition for electronic noses (E-nose). In addition to effective feature extraction, the architecture of the deep neural network plays a vital role. Currently, most applied network structures are hand-crafted by human experts, which is time-consuming and problem-dependent, making it necessary to design the structures of neural networks according to specific demands. In this work, a genetic algorithm with particle swarm optimization (GA-PSO), which possesses promising optimization capabilities, is applied to search for effective deep convolutional neural networks (CNNs) for gas classification based on E-nose technology. A novel image transformation strategy using Gramian angular field and a hybrid cost-saving method is employed in the search process, enabling adaptive and efficient CNN search on gas datasets. With the proposed methods, we can achieve an average classification accuracy of over 90 % on two public gas datasets, while also significantly reducing the model size compared to state-of-the-art CNNs. By using these novel strategies, our approach surpasses random search and basic PSO algorithm in achieving the global optimal solution, higher and more stable accuracy, and faster convergence in pattern recognition using E-nose. Our work suggests that the proposed method can quickly identify excellent CNN structures for E-nose applications.

AutoAMS: Automated attention-based multi-modal graph learning architecture search

Multi-modal attention mechanisms have been successfully used in multi-modal graph learning for various tasks. However, existing attention-based multi-modal graph learning (AMGL) architectures heavily rely on manual design, requiring huge effort and expert experience. Meanwhile, graph neural architecture search (GNAS) has made great progress toward automatically designing graph-based learning architectures. However, it is challenging to directly adopt existing GNAS methods to search for better AMGL architectures because of the search spaces that only focus on designing graph neural network architectures and the search objective that ignores multi-modal interactive information between modalities and long-term content dependencies within different modalities. To address these issues, we propose an automated attention-based multi-modal graph learning architecture search (AutoAMS) framework, which can automatically design the optimal AMGL architectures for different multi-modal tasks. Specifically, we design an effective attention-based multi-modal (AM) search space consisting of four sub-spaces, which can jointly support the automatic search of multi-modal attention representation and other components of multi-modal graph learning architecture. In addition, a novel search objective based on an unsupervised multi-modal reconstruction loss and task-specific loss is introduced to search and train AMGL architectures. The search objective can extract the global features and capture multi-modal interactions from multiple modalities. The experimental results on multi-modal tasks show strong evidence that AutoAMS is capable of designing high-performance AMGL architectures.

Adaptive multi-scale Graph Neural Architecture Search framework

Graph neural networks (GNNs) have gained significant attention for their ability to learn representations from graph-structured data, in which message passing and feature fusion strategies play an essential role. However, traditional Graph Neural Architecture Search (GNAS) mainly focuses on optimization with a static perceptive field to ease the search process. To efficiently utilize latent relationships between non-adjacent nodes as well as edge features, this work proposes a novel two-stage approach that is able to optimize GNN structures more effectively by adaptively aggregating neighborhood features in multiple scales. This adaptive multi-scale GNAS is able to assign optimal weights for different neighbors in different graphs and learning tasks. In addition, it takes latent relationships and edge features into message passing into account, and can incorporate different feature fusion strategies. Compared with traditional ones, our proposed approach can explore a much larger and more diversified search space efficiently. We also prove that traditional multi-hop GNNs are low-pass filters, which can lead to the removal of important low-frequency components of signals from remote neighbors in a graph, and they are not even expressive enough to distinguish some simple regular graphs, justifying the superiority of our approach. Experiments with seven datasets across three graph learning tasks, including graph regression, node classification, and graph classification, demonstrate that our method yields significant improvement compared with state-of-the-art GNAS approaches and human-designed GNN approaches. Specifically, for example, with our framework, the MAE of the 12-layer AM-GNAS was 0.102 for the ZINC dataset, yielding over 25% improvement.

Reinforcement Learning and Genetic Algorithm-Based Network Module for Camera-LiDAR Detection

Cameras and LiDAR sensors have been used in sensor fusion for robust object detection in autonomous driving. Object detection networks for autonomous driving are often trained again by adding or changing datasets aimed at robust performance. Repeat training is necessary to develop an efficient network module. Existing efficient network module development changes to hand design and requires much module design experience. For this, a neural architecture search was designed, but it takes much time and requires optimizing the design process. To solve this problem, we propose a two-stage optimization method for the offspring generation process in a neural architecture search based on reinforcement learning. In addition, we propose utilizing two split datasets to solve the fast convergence problem as the objective function of the genetic algorithm: source data (daytime, sunny) and target data (day/night, adversary weather). The proposed method is an efficient module generation method requiring less time than the NSGA-NET. We confirmed the performance improvement and the convergence speed reduction using the Dense dataset. Through experiments, it was proven that the proposed method generated an efficient module.

Automated machine learning-aided prediction and interpretation of gaseous by-products from the hydrothermal liquefaction of biomass

Hydrothermal liquefaction (HTL) is a thermochemical conversion technology that produces bio-oil from wet biomass without drying. However, by-product gases will inevitably be produced, and their formation is unclear. Therefore, an automated machine learning (AutoML) approach, automatically training without human intervention, was used to aid in predicting gaseous production and interpreting the formation mechanisms of four gases (CO2, CH4, CO, and H2). Specifically, four accurate optimal single-target models based on AutoML were developed with elemental compositions and HTL conditions as inputs for four gases. Herein, the gradient boosting machine (GBM) performed excellently with train R2 ≥ 0.99 and test R2 ≥ 0.80. Then, the screened GBM algorithm-based ML multi-target models (maximum average test R2 = 0.89 and RMSE = 0.39) were built to predict four gases simultaneously. Results indicated that biomass carbon, solid content, pressure, and biomass hydrogen were the top four factors for gas production from HTL of biomass. This study proposed an AutoML-aided prediction and interpretation framework, which could provide new insight for rapid prediction and revelation of gaseous compositions from the HTL process.

Hybrid kitchen safety guarding with stove fire recognition based on the Internet of Things

In this paper, we design a hybrid kitchen safety guarding framework using embedded devices and onboard sensors to detect abnormal events and block gas sources in time through the Internet of Things (IoT). According to the relevant literature we studied, this is the first framework for kitchen safety guarding that provides the following features: (1) the deep learning based model integrating densely connected convolutional networks with neural architecture search networks is developed to accurately recognize abnormal stove fire, (2) the acceleration correction method is designed to correct the sensed accelerometer values for estimating the actual earthquake level, and (3) the proper gas leakage threshold is defined to precisely detect the gas leakage for automatic gas blocking, and remote surveillance and control are provided to monitor the kitchen environment and control the gas source anytime and anywhere. In particular, an Android-based prototype consisting of the IoT device, diverse sensors, dedicated server, and smartphones is implemented to verify the feasibility and superiority of our framework. Experimental results show that our framework outperforms existing methods and can precisely recognize stove fire intensity and detect earthquake levels for kitchen safety guarding.

Integration between constrained optimization and deep networks: a survey.

Integration between constrained optimization and deep networks has garnered significant interest from both research and industrial laboratories. Optimization techniques can be employed to optimize the choice of network structure based not only on loss and accuracy but also on physical constraints. Additionally, constraints can be imposed during training to enhance the performance of networks in specific contexts. This study surveys the literature on the integration of constrained optimization with deep networks. Specifically, we examine the integration of hyper-parameter tuning with physical constraints, such as the number of FLOPS (FLoating point Operations Per Second), a measure of computational capacity, latency, and other factors. This study also considers the use of context-specific knowledge constraints to improve network performance. We discuss the integration of constraints in neural architecture search (NAS), considering the problem as both a multi-objective optimization (MOO) challenge and through the imposition of penalties in the loss function. Furthermore, we explore various approaches that integrate logic with deep neural networks (DNNs). In particular, we examine logic-neural integration through constrained optimization applied during the training of NNs and the use of semantic loss, which employs the probabilistic output of the networks to enforce constraints on the output.

Multi-objective Evolutionary Neural Architecture Search for Recurrent Neural Networks

Artificial neural network (NN) architecture design is a nontrivial and time-consuming task that often requires a high level of human expertise. Neural architecture search (NAS) serves to automate the design of NN architectures and has proven to be successful in automatically finding NN architectures that outperform those manually designed by human experts. NN architecture performance can be quantified based on multiple objectives, which include model accuracy and some NN architecture complexity objectives, among others. The majority of modern NAS methods that consider multiple objectives for NN architecture performance evaluation are concerned with automated feed forward NN architecture design, which leaves multi-objective automated recurrent neural network (RNN) architecture design unexplored. RNNs are important for modeling sequential datasets, and prominent within the natural language processing domain. It is often the case in real world implementations of machine learning and NNs that a reasonable trade-off is accepted for marginally reduced model accuracy in favour of lower computational resources demanded by the model. This paper proposes a multi-objective evolutionary algorithm-based RNN architecture search method. The proposed method relies on approximate network morphisms for RNN architecture complexity optimisation during evolution. The results show that the proposed method is capable of finding novel RNN architectures with comparable performance to state-of-the-art manually designed RNN architectures, but with reduced computational demand.

Deep Learning and Neural Architecture Search for Optimizing Binary Neural Network Image Super Resolution.

The evolution of super-resolution (SR) technology has seen significant advancements through the adoption of deep learning methods. However, the deployment of such models by resource-constrained devices necessitates models that not only perform efficiently, but also conserve computational resources. Binary neural networks (BNNs) offer a promising solution by minimizing the data precision to binary levels, thus reducing the computational complexity and memory requirements. However, for BNNs, an effective architecture is essential due to their inherent limitations in representing information. Designing such architectures traditionally requires extensive computational resources and time. With the advancement in neural architecture search (NAS), differentiable NAS has emerged as an attractive solution for efficiently crafting network structures. In this paper, we introduce a novel and efficient binary network search method tailored for image super-resolution tasks. We adapt the search space specifically for super resolution to ensure it is optimally suited for the requirements of such tasks. Furthermore, we incorporate Libra Parameter Binarization (Libra-PB) to maximize information retention during forward propagation. Our experimental results demonstrate that the network structures generated by our method require only a third of the parameters, compared to conventional methods, and yet deliver comparable performance.

A novel blood-based epigenetic biosignature in first-episode schizophrenia patients through automated machine learning

Schizophrenia (SCZ) is a chronic, severe, and complex psychiatric disorder that affects all aspects of personal functioning. While SCZ has a very strong biological component, there are still no objective diagnostic tests. Lately, special attention has been given to epigenetic biomarkers in SCZ. In this study, we introduce a three-step, automated machine learning (AutoML)-based, data-driven, biomarker discovery pipeline approach, using genome-wide DNA methylation datasets and laboratory validation, to deliver a highly performing, blood-based epigenetic biosignature of diagnostic clinical value in SCZ. Publicly available blood methylomes from SCZ patients and healthy individuals were analyzed via AutoML, to identify SCZ-specific biomarkers. The methylation of the identified genes was then analyzed by targeted qMSP assays in blood gDNA of 30 first-episode drug-naïve SCZ patients and 30 healthy controls (CTRL). Finally, AutoML was used to produce an optimized disease-specific biosignature based on patient methylation data combined with demographics. AutoML identified a SCZ-specific set of novel gene methylation biomarkers including IGF2BP1, CENPI, and PSME4. Functional analysis investigated correlations with SCZ pathology. Methylation levels of IGF2BP1 and PSME4, but not CENPI were found to differ, IGF2BP1 being higher and PSME4 lower in the SCZ group as compared to the CTRL group. Additional AutoML classification analysis of our experimental patient data led to a five-feature biosignature including all three genes, as well as age and sex, that discriminated SCZ patients from healthy individuals [AUC 0.755 (0.636, 0.862) and average precision 0.758 (0.690, 0.825)]. In conclusion, this three-step pipeline enabled the discovery of three novel genes and an epigenetic biosignature bearing potential value as promising SCZ blood-based diagnostics.

Deep learning for geological mapping in the overburden area

This paper aims to achieve bedrock geologic mapping in the overburden area using big data, distributed computing, and deep learning techniques. First, the satellite Bouguer gravity anomaly with a resolution of 2′×2′ in the range of E66°-E96°, N40°-N55° and 1:5000000 Asia-European geological map are used to design a dataset for bedrock prediction. Then, starting from the gravity anomaly formula in the spherical coordinate system, we deduce the non-linear functional between rock density ρ and rock mineral composition m, content p, buried depth h, diagenesis time t and other variables. We analyze the feasibility of using deep neural network to approximate the above nonlinear generalization. The problem of solving deep neural network parameters is transformed into a non-convex optimization problem. We give an iterative, gradient descent-based solution algorithm for the non-convex optimization problem. Utilizing neural architecture search (NAS) and human-designed approach, we propose a geological-geophysical mapping network (GGMNet). The dataset for the network consists of both gravity anomaly and a priori geological information. The network has fast convergence speed and stable iteration during the training process. It also has better performance than a single neural network search or human-designed architectures, with the mean pixel accuracy (MAP) = 63.1% and the frequency weighted intersection over union (FWIoU) = 42.88. Finally, the GGMNet is used to predict the rock distribution of the Junggar Basin.

Sequential node search for faster neural architecture search

Neural Architecture Search (NAS) has progressed significantly by reducing search costs since it was first introduced. The faster NAS algorithms use cell-type architectures. The cells are made up of nodes, and typically, all the node operations are searched together. This paper proposes a novel algorithm that searches the nodes of the cell type NAS independently of one another. This is done in a sequential manner, hence named SeQuential Neural Architecture Search (SQNAS). It is hypothesized that the hierarchical nature of nodes’ feature tensors is the proper guide to organize the sequence of the nodes. SQNAS reduces the size of the super graph search space at any instant of search, thus reducing the search cost considerably. The independence assumption is justified considering the competitive performance obtained. SQNAS search is also done in cycles of several restarts for better regularization. With a single NVIDIA GTX 1080Ti GPU, the search is completed in 6 h for the image classification task, obtaining a 2.55% test error on CIFAR10. For the Convolutional Neural Network (CNN) and Recurrent Neural Network (RNN) cells searched on smaller datasets, the transferability to deeper models with larger datasets is tested with ImageNet and WikiText-2. Partial channel SQNAS was also implemented, which completed the search in just 1.5 h on the CIFAR10 dataset with competitive test results. This is one of the fastest searches achieved on CIFAR10.

Development and validation of an automatic machine learning model to predict abnormal increase of transaminase in valproic acid-treated epilepsy.

Valproic acid (VPA) is a primary medication for epilepsy, yet its hepatotoxicity consistently raises concerns among individuals. This study aims to establish an automated machine learning (autoML) model for forecasting the risk of abnormal increase of transaminase levels while undergoing VPA therapy for 1995 epilepsy patients. The study employed the two-tailed T test, Chi-square test, and binary logistic regression analysis, selecting six clinical parameters, including age, stature, leukocyte count, Total Bilirubin, oral dosage of VPA, and VPA concentration. These variables were used to build a risk prediction model using "H2O" autoML platform, achieving the best performance (AUC training = 0.855, AUC test = 0.789) in the training and testing data set. The model also exhibited robust accuracy (AUC valid = 0.742) in an external validation set, underscoring its credibility in anticipating VPA-induced transaminase abnormalities. The significance of the six variables was elucidated through importance ranking, partial dependence, and the TreeSHAP algorithm. This novel model offers enhanced versatility and explicability, rendering it suitable for clinicians seeking to refine parameter adjustments and address imbalanced data sets, thereby bolstering classification precision. To summarize, the personalized prediction model for VPA-treated epilepsy, established with an autoML model, displayed commendable predictive capability, furnishing clinicians with valuable insights for fostering pharmacovigilance.

Multimodal deep learning for dementia classification using text and audio

Dementia is a progressive neurological disorder that affects the daily lives of older adults, impacting their verbal communication and cognitive function. Early diagnosis is important to enhance the lifespan and quality of life for affected individuals. Despite its importance, diagnosing dementia is a complex process. Automated machine learning solutions involving multiple types of data have the potential to improve the process of automated dementia screening. In this study, we build deep learning models to classify dementia cases from controls using the Pitt Cookie Theft dataset from DementiaBank, a database of short participant responses to the structured task of describing a picture of a cookie theft. We fine-tune Wav2vec and Word2vec baseline models to make binary predictions of dementia from audio recordings and text transcripts, respectively. We conduct experiments with four versions of the dataset: (1) the original data, (2) the data with short sentences removed, (3) text-based augmentation of the original data, and (4) text-based augmentation of the data with short sentences removed. Our results indicate that synonym-based text data augmentation generally enhances the performance of models that incorporate the text modality. Without data augmentation, models using the text modality achieve around 60% accuracy and 70% AUROC scores, and with data augmentation, the models achieve around 80% accuracy and 90% AUROC scores. We do not observe significant improvements in performance with the addition of audio or timestamp information into the model. We include a qualitative error analysis of the sentences that are misclassified under each study condition. This study provides preliminary insights into the effects of both text-based data augmentation and multimodal deep learning for automated dementia classification.

MSR63 Improving Efficiency in Analysis of Real-World Data with an Automated Machine Learning Tool

MSR63 Improving Efficiency in Analysis of Real-World Data with an Automated Machine Learning Tool

A framework of a data-driven model for ship performance

An accurate assessment of a ship's required power is increasingly relevant for ship operations. We used a simplified framework of a data-driven model to predict ship's fuel consumption. Our approach was based on the learning capabilities of a generalized AutoML (Automated Machine Learning) process, trained with a variety of databases obtained from Computational-Fluid-Dynamics (CFD) simulations or from simplified numerical methods. These CFD simulations were conducted by solving the Reynolds Averaged Navier-Stokes (RANS) equations, using the STARCCM + commercial CFD software to calculate the ship resistance at speed under different operating conditions. Initially, we conducted a statistical analysis to select the independent variables before fitting the regression models and identifying potentially wrong assumptions. The AutoML process allowed optimizing the model's hyperparameters and designing the topology of the neural networks. For a set of unknown scenarios, comparative predictions obtained from the data-driven model and from numerical simulations showed that the data-driven model, trained with results obtained from CFD simulations, accurately and efficiently predicted ship operational parameters under realistic operating conditions, thereby dispensing with the need to perform elaborate CFD computations. Specifically, this low-cost and efficient operational data-driven technique forecasted the ship's operational fuel consumption although only a limited amount of recorded operational data was available.

Development and validation of an automated machine learning model for the multi-class classification of diabetic retinopathy, central retinal vein occlusion and branch retinal vein occlusion based on color fundus photographs

IntroductionAutomated machine learning (AutoML) is a novel artificial intelligence (AI) strategy that enables clinicians without coding experience to develop their own AI models. This study assessed the discriminative performance of AutoML in differentiating diabetic retinopathy (DR), central retinal vein occlusion (CRVO) and branch retinal vein occlusion (BRVO) from normal fundi using color fundus photographs (CFPs). MethodsWe carried out AutoML model design using CFPs retrieved from a publicly available CFP data set (3200 labelled images). The retrieved CFPs were reviewed for quality and then uploaded to the Google Cloud Vertex AI platform for AutoML training and testing. We trained a multi-class classification model to differentiate DR, CRVO, BRVO from normal fundi using 875 CFPs and externally validated the model using 210 CFPs obtained from another dataset. Performance metrics, including area under receiver operator curve (AUROC) and sensitivity were reported. We then compared the AutoML model to state-of-the-art deep learning (DL)-based DR and RVO multi-class models identified through a literature search. ResultsOur AutoML model showed high discriminative performance in the multi-class classification of DR, CRVO and BRVO based on CFPs, with an AUROC, precision and recall reaching 0.995, 95.4% and 95.4% respectively at the 0.5 confidence threshold. The per-label sensitivity and specificity, respectively, were normal fundi (97.5%, 100%), DR (100%, 93.88%), CRVO (66.67%, 100%) and BRVO (71.43%, 98.73%). Our AutoML model generally showed similar performance to the state-of-the-art DL classifiers. ConclusionOur AutoML model can detect DR, CRVO, and BRVO in CFPs with good diagnostic accuracy and is a potentially useful screening tool.