Architecture Search Research Articles

Towards Automated Model Design on Recommender Systems

The increasing popularity of deep learning models has created new opportunities for developing AI-based recommender systems. Designing recommender systems using deep neural networks requires careful architecture design, and further optimization demands extensive co-design efforts on jointly optimizing model architecture and hardware. Design automation, such as Automated Machine Learning (AutoML), is necessary to fully exploit the potential of recommender model design, including model choices and model-hardware co-design strategies. We introduce a novel paradigm that utilizes weight sharing to explore abundant solution spaces. Our paradigm creates a large supernet to search for optimal architectures and co-design strategies to address the challenges of data multi-modality and heterogeneity in the recommendation domain. From a model perspective, the supernet includes a variety of operators, dense connectivity, and dimension search options. From a co-design perspective, it encompasses versatile Processing-In-Memory (PIM) configurations to produce hardware-efficient models. Our solution space’s scale, heterogeneity, and complexity pose several challenges, which we address by proposing various techniques for training and evaluating the supernet. Our crafted models show promising results on three Click-Through Rates (CTR) prediction benchmarks, outperforming both manually designed and AutoML-crafted models with state-of-the-art performance when focusing solely on architecture search. From a co-design perspective, we achieve 2 × FLOPs efficiency, 1.8 × energy efficiency, and 1.5 × performance improvements in recommender models.

Adaptive Neural Architecture Search Using Meta-Heuristics: Discovering Fine-Tuned Predictive Models for Photocatalytic CO2 Reduction

This study aims to contribute to the reduction of carbon dioxide and the production of hydrogen through an investigation of the photocatalytic reaction process. Machine learning algorithms can be used to predict the hydrogen yield in the photocatalytic carbon dioxide reduction process. Although regression-based approaches provide good results, the accuracy achieved with classification algorithms is not very high. In this context, this study presents a new method, Adaptive Neural Architecture Search (NAS) using metaheuristics, to improve the capacity of ANNs in estimating the hydrogen yield in the photocatalytic carbon dioxide reduction process through classification. The NAS process was carried out with a tool named HyperNetExplorer, which was developed with the aim of finding the ANN architecture providing the best prediction accuracy through changing ANN hyperparameters, such as the number of layers, number of neurons in each layer, and the activation functions of each layer. The nature of the NAS process in this study was adaptive, since the process was accomplished through optimization algorithms. The ANNs discovered with HyperNetExplorer demonstrated significantly higher prediction performance than the classical ML algorithms. The results indicated that the NAS helped to achieve better performance in the estimation of the hydrogen yield in the photocatalytic carbon dioxide reduction process.

Deep Learning-Based Network Intrusion Detection Systems

With the rapid growth of the internet, the security threats to computer networks have escalated significantly, making the reduction and prevention of cybercrime a top priority in the digital age. Traditional Network Intrusion Detection Systems (NIDS) struggle with limitations in detection accuracy and real-time performance as attackers employ increasingly sophisticated techniques. In recent years, deep learning has emerged as a prominent solution in the NIDS field due to its powerful capabilities in feature extraction and classification. This paper reviews the application of deep learning in NIDS, with a focus on Convolutional Neural Networks (CNN), Long Short-Term Memory Networks (LSTM), and their hybrid models. The paper discusses the strengths of these models in capturing spatial and temporal features and examines their performance on key datasets such as KDD Cup 99 and UNSW-NB15. Additionally, the paper addresses challenges related to computational complexity, real-time performance, and model interpretability, while suggesting future research directions, including model optimization, lightweight architectures, and improved interpretability. Finally, the potential of Automated Machine Learning (AutoML) in advancing NIDS design and enhancing response capabilities is explored. This study offers valuable insights for further research and development in NIDS.

Analysis of Different Methods in Pedestrian Re-identification

This study focuses on the field of pedestrian re-identification (ReID), aiming to enhance the accuracy and efficiency of individual recognition through advanced deep learning models. The article first introduces two deep learning models: Omni-Scale Feature Learning for Person Re-Identification (OSNet) and Multi-Scale Interaction Network (MSINet). OSNet employs Depthwise Separable Convolutions (DSC) and Omni-Scale Residual Blocks to improve the ability to learn features across different scales. MSINet, on the other hand, utilizes neural architecture search technology to design a lightweight network architecture, enhancing feature discrimination and flexible utilization through Twins Contrastive Mechanism (TCM) and Multi-Scale Interaction (MSI). Additionally, a Spatial Alignment Module (SAM) is proposed to enhance the consistency of images under different viewpoints or conditions. The experimental section selects two widely used pedestrian re-identification datasets, Market1501 and MSMT17, for evaluation, and the results show that MSINet outperforms existing methods in terms of accuracy and stability. The article concludes by summarizing the advantages of OSNet and MSINet in multi-scale feature learning and points out their application limitations.

TransNAS-TSAD: harnessing transformers for multi-objective neural architecture search in time series anomaly detection

TransNAS-TSAD: harnessing transformers for multi-objective neural architecture search in time series anomaly detection

Chain-structured neural architecture search for financial time series forecasting

AbstractNeural architecture search (NAS) emerged as a way to automatically optimize neural networks for a specific task and dataset. Despite an abundance of research on NAS for images and natural language applications, similar studies for time series data are lacking. Among NAS search spaces, chain-structured are the simplest and most applicable to small datasets like time series. We compare three popular NAS strategies on chain-structured search spaces: Bayesian optimization (specifically Tree-structured Parzen Estimator), the hyperband method, and reinforcement learning in the context of financial time series forecasting. These strategies were employed to optimize simple well-understood neural architectures like the MLP, 1D CNN, and RNN, with more complex temporal fusion transformers (TFT) and their own optimizers included for comparison. We find Bayesian optimization and the hyperband method performing best among the strategies, and RNN and 1D CNN best among the architectures, but all methods were very close to each other with a high variance due to the difficulty of working with financial datasets. We discuss our approach to overcome the variance and provide implementation recommendations for future users and researchers.

EGNAS: Efficient Graph Neural Architecture Search Through Evolutionary Algorithm

The primary objective of our research is to enhance the efficiency and effectiveness of Neural Architecture Search (NAS) with regard to Graph Neural Networks (GNNs). GNNs have emerged as powerful tools for learning from unstructured network data, compensating for several known limitations of Convolutional Neural Networks (CNNs). However, the automatic search for optimal GNN architectures has seen little progressive advancement so far. To address this gap, we introduce the Efficient Graph Neural Architecture Search (EGNAS), a method that leverages the advantages of evolutionary search strategies. EGNAS incorporates inherited parameter sharing, allowing offspring to inherit parameters from their parents, and utilizes half epochs to improve optimization stability. In addition, EGNAS employs a combined evolutionary search, which explores both the model structure and the hyperparameters within a large search space, resulting in improved performance. Our experimental results demonstrate that EGNAS outperforms state-of-the-art methods in node classification tasks on the Cora, Citeseer, and PubMed datasets while maintaining a high degree of computational efficiency. In particular, EGNAS is the fastest GNN architecture search method in terms of search time, particularly when compared to precedently suggested evolutionary search strategies, delivering performance up to 40 times faster.

Adaptive evolutionary neural architecture search based on one-dimensional convolutional neural network for electric rudder fault diagnosis

Abstract The electric rudder is the core actuator of the flight control system. Fault diagnosis of rudders is essential for the production and repair of rudders. While existing methods for rudder fault diagnosis are effective, the manual design of neural network models is a time-consuming and challenging process. Therefore, this paper proposes a fault diagnosis framework for the electric rudder based on an adaptive evolutionary neural architecture search (AENAS-FD). AENAS-FD employs an adaptive strategy to guide the evolution of a one-dimensional convolutional neural network towards achieving optimal diagnostic accuracy. This adaptive strategy adjusts the relevant parameters of the genetic operator based on the relationship between individual and population fitness. This leads to improved algorithm search performance and mitigates premature convergence. The experiments on the real electric rudder dataset demonstrate that AENAS-FD can generate superior network architectures for diagnosing rudder faults, exhibiting better diagnostic accuracy when compared to manually designed networks.

Automated Machine Learning: A Survey of Tools and Techniques

Automated Machine Learning (AutoML) has revolutionized the field of machine learning by automating complex and time-intensive tasks such as data preprocessing, model selection, and hyperparameter tuning. This study explores the capabilities, limitations, and practical applications of six widely used AutoML tools: Auto-sklearn, TPOT, H2O.ai, Google Cloud AutoML, Microsoft Azure AutoML, and Amazon SageMaker Autopilot. By evaluating these tools across diverse datasets—spanning tabular data, time series, image classification, and text sentiment analysis—the research highlights their predictive performance, computational efficiency, scalability, and explainability. Proprietary tools demonstrated superior scalability and efficiency through cloud integration, while open-source platforms provided more interpretability and flexibility. However, challenges such as lack of transparency in advanced neural architecture search mechanisms and ethical considerations, including bias mitigation, remain prevalent. This study concludes that while AutoML tools significantly lower the barrier to entry for machine learning, ongoing advancements are required to balance performance, usability, and ethical standards, making AutoML an integral solution for real-world applications.

Numerical analysis and data-driven optimization of energy harvesting efficiency from wake-induced vibration in a multi-cylinder configuration

Wake-induced vibrations (WIV) in multi-cylinder configurations have demonstrated greater energy harvesting efficiency in hydrokinetic applications compared to conventional vortex-induced vibrations (VIV) of a single cylinder. However, the complex fluid-structure interactions make it challenging to identify optimal configurations for maximum power output, as extensive simulations across numerous parameter combinations lead to substantial computational costs with traditional computational fluid dynamics (CFD) methods. To address this challenge, we developed a data-driven model using automated machine learning (AutoML) techniques, focusing on four key parameters: spacing, diameter, damping, and reduced velocity. Trained on comprehensive datasets from validated CFD simulations, this model integrates multiple algorithms to predict the power efficiency of WIV systems with high accuracy. Our approach enables rapid and precise evaluations of power efficiency across a broad range of configurations, significantly reducing the computational burden compared to traditional CFD approaches. The results indicate that optimal configurations, characterized by larger upstream cylinder diameters, higher damping ratios, and ideal spacing ratios, can achieve power generation efficiencies of up to 59.15%. Further analysis of vorticity contours reveals that synchronized interactions between upstream vortex shedding and downstream structure motions enhance WIV, thereby improving energy harvesting efficiency.

ICESat-2 data denoising and forest canopy height estimation using Machine Learning

Supervised classification methods can distinguish between noise and signal in ice, cloud, and land elevation satellite-2 (ICESat-2) data across various feature perspectives and autonomously optimize parameters. Nevertheless, model generalization remains a significant limitation for practical applications. This study focuses on developing a universal denoising model for ICESat-2 using machine learning algorithms and analyzing its spatial transferability under various forest and terrain conditions. A photon-denoising feature parameter system is developed based on the analysis of the three-dimensional distribution of photons in forested regions. This system reduces the parameters dependent on absolute physical quantities and increases those that are less influenced by terrain and forest features to enhance the model’s transferability. Subsequently, automated machine learning algorithms (AutoML) are used for model selection and parameter optimization across six non-parametric regression models. We evaluate the accuracies of the local, global, and transfer models in estimating canopy height across four representative forested areas in China. Results show that the algorithm can effectively distinguish between signal and noise photons. The estimated canopy heights from signal photons are highly consistent with heights obtained using airborne laser scanning (ALS), exhibiting a Pearson correlation coefficient (r) of 0.89, root mean square errors (RMSE) of 3.75 m, relative root mean square error (rRMSE) of 0.27, relative bias (rBias) of −0.11 and mean Bias of −1.45 m. Notably, the accuracy of canopy height estimation by the global model has increased by an average of 21 % compared to ICESat-2 land-vegetation along-track products (ATL08). Furthermore, the model exhibits significant spatial transfer capabilities, with the accuracies of the transfer model exceeding those of ATL08 by margins ranging from 4 % to 41 %. This study marks a significant advancement in photon-denoising methodologies, providing a robust and transferable solution for large-scale environmental data analysis.

Construction and evaluation of prediction model for postoperative re-fractures in elderly patients with hip fractures

ObjectiveThe aim of study was to construct a postoperative re-fracture prediction model for elderly hip fracture patients using an automated machine learning algorithm to provide a basis for early identification of patients with high risk of re-fracture occurrence. MethodsClinical data were collected and subjected to univariate and multivariate analyses to determine the independent risk factors affecting postoperative re-fracture of hip fracture in the elderly. The collected data were divided into training and validation sets in a ratio of 7:3, AutoGluon was applied to construct LightGBMXT, LightGBM, RandomForestGini, RandomForestEntr, CatBoost, NeuralNetFastAI, XGBoost, NeuralNetTorch, LightGBMLarge and WeightedEnsemble_L2 prediction models, and the constructed models were evaluated using evaluation indicators. The models were externally validated and the model with the best prediction performance was selected. ResultsThe incidence of postoperative re-fracture was about 11.7%, and age, comorbid diabetes mellitus, comorbid osteoporosis, rehabilitation exercise status, and preoperative total protein level were considered as independent risk factors. The top three models in terms of AUC values in the training set were WeightedEnsemble_L2 (0.9671), XGBoost (0.9636), and LightGBM (0.9626), the WeightedEnsemble_L2 (0.9759) was best in the external validation. Based on the AUC and other evaluation indicators, WeightedEnsemble_L2 was considered the model with the best prediction performance. ConclusionThe constructed model is highly generalizable and applicable, and can be used as an effective tool for healthcare professionals to assess and manage patients’ risk of re-fracture.

Searching to Exploit Memorization Effect in Deep Learning With Noisy Labels.

Sample selection approaches are popular in robust learning from noisy labels. However, how to control the selection process properly so that deep networks can benefit from the memorization effect is a hard problem. In this paper, motivated by the success of automated machine learning (AutoML), we propose to control the selection process by bi-level optimization. Specifically, we parameterize the selection process by exploiting the general patterns of the memorization effect in the upper-level, and then update these parameters using predicting accuracy obtained from model training in the lower-level. We further introduce semi-supervised learning algorithms to utiilize noisy-labeled data as unlabeled data. To solve the bi-level optimization problem efficiently, we consider more information from the validation curvature by the Newton method and cubic regularization method. We provide convergence analysis for both optimization methods. Results show that while both methods can converge to an (approximately) stationary point, the cubic regularization method can find better local optimal than the Newton method with less time. Experiments on both benchmark and real-world data sets demonstrate that the proposed searching method can lead to significant improvements upon existing methods. Compared with existing AutoML approaches, our method is much more efficient on finding a good selection schedule.

A Comparative Analysis of Automated Machine Learning Libraries for Electricity Price Forecasting

Abstract Reliable and accurate electricity price forecasting algorithms can be used to inform efficient energy consumption schedules and maximise profits for electricity traders. Operating within Ireland’s Integrated Single Electricity Market (I-SEM), traders can buy and sell electricity at fluctuating hourly rates whose day-ahead prices are published at approximately 13:00 GMT day-1. Access to electricity price predictions earlier than this publication time allows stakeholders an expanded timeframe to facilitate energy cost-aware scheduling. While many studies have been conducted to espouse various machine learning and statistical approaches to electricity price forecasting, these models tend to be bespoke and require in-depth knowledge regarding model implementation. The problem of requiring such expertise is not unique to time series forecasting, and research into mitigating such limitations exists in the form of Automated Machine Learning (AutoML). AutoML aims to derive effective models while automating various steps typically required for machine learning experimentation, such as pre-processing, model selection, validation, etc. Given the increasing proliferation of AutoML tools and frameworks, this paper applies eight Python-based AutoML libraries to day-ahead electricity price forecasting on an excerpt of I-SEM data. These libraries are compared across a series of error metrics and training times to produce an empirical benchmark that can be utilised to select high-performing AutoML tools for further price forecasting research and other forms of time series forecasting. AutoKeras is found to produce accurate forecasts but requires careful configuration to avoid long runtimes. PyCaret, Ludwig, FLAML and FEDOT also generate favourable results while being significantly easier to configure.

An effective surrogate-assisted rank method for evolutionary neural architecture search

Evolutionary neural architecture search (ENAS) is able to automatically design high-performed architectures of deep neural networks (DNNs) for specific tasks. In recent years, surrogate models have gained significant traction because they can estimate the performance of neural architectures, avoiding excessive computational costs for training. However, most existing surrogate models primarily predict the performance of architectures directly or predict pairwise comparison relationships, which makes it challenging to obtain the rank of a group of architectures when training samples are limited. To address this problem, we propose TCMR-ENAS, an effective triple-competition model-assisted rank method for ENAS. TCMR-ENAS employs a novel triple-competition surrogate model combined with a score-based fitness evaluation method to predict group performance rank. Moreover, a progressive online learning method is proposed to enhance the predictive performance of the triple-competition surrogate model in the framework of modified genetic search. To validate the effectiveness of TCMR-ENAS, we conducted a series of experiments on NAS-Bench-101, NAS-Bench-201, NATS-Bench and NAS-Bench-301, respectively. Experimental results show that TCMR-ENAS can achieve better performance with lower computational resources. The accuracies of searched architectures achieve the best results compared with those of the state-of-the-art methods with limited training samples. In addition, the factors that may influence the effectiveness of TCMR-ENAS are explored in the ablation studies.

Divert More Attention to Vision-Language Object Tracking.

Multimodal vision-language (VL) learning has noticeably pushed the tendency toward generic intelligence owing to emerging large foundation models. However, tracking, as a fundamental vision problem, surprisingly enjoys less bonus from recent flourishing VL learning. We argue that the reasons are two-fold: the lack of large-scale vision-language annotated videos and ineffective vision-language interaction learning of current works. These nuisances motivate us to design more effective vision-language representation for tracking, meanwhile constructing a large database with language annotation for model learning. Particularly, in this paper, we first propose a general attribute annotation strategy to decorate videos in six popular tracking benchmarks, which contributes a large-scale vision-language tracking database with more than 23,000 videos. We then introduce a novel framework to improve tracking by learning a unified-adaptive VL representation, where the cores are the proposed asymmetric architecture search and modality mixer (ModaMixer). To further improve VL representation, we introduce a contrastive loss to align different modalities. To thoroughly evidence the effectiveness of our method, we integrate the proposed framework on three tracking methods with different designs, i.e., the CNN-based SiamCAR (Guo et al. 2020), the Transformer-based OSTrack (Ye et al. 2022), and the hybrid structure TransT (Chen et al. 2021). The experiments demonstrate that our framework can significantly improve all baselines on six benchmarks. Besides empirical results, we theoretically analyze our approach to show its rationality. By revealing the potential of VL representation, we expect the community to divert more attention to VL tracking and hope to open more possibilities for future tracking with diversified multimodal messages.

Data race detection via few-shot parameter-efficient fine-tuning

The OpenMP programming model is playing an increasing role in parallelization on shared-memory systems owing to its simplicity of operation and portability. OpenMP provides the semantic equivalent of a parallel program for the original sequential program. Though it is easier to write parallel programs using OpenMP, writing them correctly is a challenge. Data race conditions errors can easily occur during the writing process, particularly by inexperienced programmers. Some data race checkers have been developed to help programmers check for data race in parallel programs. However, several of them have constraints on the input and thread configuration, time overhead, and scope of program analysis. In this study, we target data race detection in OpenMP parallel programs to address the issues of constraints from checkers. We propose a few-shot parameter-efficient fine-tuning method using adapter module to address data race detection issue. The proposed method does not require a large labeled dataset, and it makes data efficient. A generic dataset is constructed with a limited number of labeled data, containing diverse OpenMP patterns for data race detection. A neural architecture search approach is employed to improve the performance of detection. The experimental results on the generated and open-source datasets demonstrate that our method is effective and improves race detection compared with traditional methods.

Zero-Shot Neural Architecture Search: Challenges, Solutions, and Opportunities.

Recently, zero-shot (or training-free) Neural Architecture Search (NAS) approaches have been proposed to liberate NAS from the expensive training process. The key idea behind zero-shot NAS approaches is to design proxies that can predict the accuracy of some given networks without training the network parameters. The proxies proposed so far are usually inspired by recent progress in theoretical understanding of deep learning and have shown great potential on several datasets and NAS benchmarks. This paper aims to comprehensively review and compare the state-of-the-art (SOTA) zero-shot NAS approaches, with an emphasis on their hardware awareness. To this end, we first review the mainstream zero-shot proxies and discuss their theoretical underpinnings. We then compare these zero-shot proxies through large-scale experiments and demonstrate their effectiveness in both hardware-aware and hardware-oblivious NAS scenarios. Finally, we point out several promising ideas to design better proxies.

Knowledge-aware evolutionary graph neural architecture search

Graph neural architecture search (GNAS) can customize high-performance graph neural network architectures for specific graph tasks or datasets. However, existing GNAS methods begin searching for architectures from a zero-knowledge state, ignoring the prior knowledge that may improve the search efficiency. The available knowledge base (e.g. NAS-Bench-Graph) contains many rich architectures and their multiple performance metrics, such as the accuracy (#Acc) and number of parameters (#Params). This study proposes exploiting such prior knowledge to accelerate the multi-objective evolutionary search on a new graph dataset, named knowledge-aware evolutionary GNAS (KEGNAS). KEGNAS employs the knowledge base to train a knowledge model and a deep multi-output Gaussian process (DMOGP) in one go, which generates and evaluates transfer architectures in only a few GPU seconds. The knowledge model first establishes a dataset-to-architecture mapping, which can quickly generate candidate transfer architectures for a new dataset. Subsequently, the DMOGP with architecture and dataset encodings is designed to predict multiple performance metrics for candidate transfer architectures on the new dataset. According to the predicted metrics, non-dominated candidate transfer architectures are selected to warm-start the multi-objective evolutionary algorithm for optimizing the #Acc and #Params on a new dataset. Empirical studies on NAS-Bench-Graph and five real-world datasets show that KEGNAS swiftly generates top-performance architectures, achieving 4.27% higher accuracy than advanced evolutionary baselines and 11.54% higher accuracy than advanced differentiable baselines. In addition, ablation studies demonstrate that the use of prior knowledge significantly improves the search performance.

AutoLDT: a lightweight spatio-temporal decoupling transformer framework with AutoML method for time series classification

Time series classification finds widespread applications in civil, industrial, and military fields, while the classification performance of time series models has been improving with the recent development of deep learning. However, the issues of feature extraction effectiveness, model complexity, and model design uncertainty constrain the further development of time series classification. To address the above issues, we propose a Lightweight Spatio-Temporal Decoupling Transformer framework based on Automated Machine Learning technique (AutoLDT). The framework proposes a novel lightweight Transformer with fuzzy position encoding, TS-separable linear self-attention mechanism, and convolutional feedforward network, which mine the temporal and spatial features, as well as the local and global relationship of time series. Fuzzy positional encoding integrates fuzzy ideas to enhance the generalization performance of model information mining. TS-separable linear self-attention mechanism and convolutional feedforward network achieve feature extraction in a lightweight way by decoupling temporal and spatial features of time series. Notably, we adopt the Covariance Matrix Adaptation Evolution Strategy and global adaptive pruning technique to realize automated network structure design, which further improves the model training efficiency and automation, and avoids the uncertainty problem of network design. Finally, we validate the effectiveness of the proposed framework on publicly available UCR and UEA time series datasets. The experimental results show that the proposed framework not only improves the model classification performance in a lightweight way but also dramatically improves the model training efficiency.