Architecture Search Research Articles

Hga-lstm: LSTM architecture and hyperparameter search by hybrid GA for air pollution prediction

Hga-lstm: LSTM architecture and hyperparameter search by hybrid GA for air pollution prediction

Few-shot learning to identify atypical endometrial hyperplasia and endometrial cancer based on transvaginal ultrasonic images

ObjectiveIt is challenging to accurately distinguish atypical endometrial hyperplasia (AEH) and endometrial cancer (EC) under routine transvaginal ultrasonic (TVU) detection. Our research aims to use the few-shot learning (FSL) method to identify non-atypical endometrial hyperplasia (NAEH), AEH, and EC based on limited TVU images. MethodsThe TVU images of pathologically confirmed NAEH, AEH, and EC patients (n = 33 per class) were split into the support set (SS, n = 3 per class) and the query set (QS, n = 30 per class). Next, we used dual pretrained ResNet50 V2 which pretrained on ImageNet first and then on extra collected TVU images to extract 1*64 eigenvectors from the TVU images in SS and QS. Then, the Euclidean distances were calculated between each TVU image in QS and nine TVU images of SS. Finally, the k-nearest neighbor (KNN) algorithm was used to diagnose the TVU images in QS. ResultsThe overall accuracy and macro precision of the proposed FSL model in QS were 0.878 and 0.882 respectively, superior to the automated machine learning models, traditional ResNet50 V2 model, junior sonographer, and senior sonographer. When identifying EC, the proposed FSL model achieved the highest precision of 0.964, the highest recall of 0.900, and the highest F1-score of 0.931. ConclusionsThe proposed FSL model combining dual pretrained ResNet50 V2 eigenvectors extractor and KNN classifier presented well in identifying NAEH, AEH, and EC patients with limited TVU images, showing potential in the application of computer-aided disease diagnosis.

An Automated Machine Learning-Based Quantitative Multiparametric Approach for Mitral Regurgitation Severity Grading.

Considering the high prevalence of mitral regurgitation (MR) and the highly subjective, variable MR severity reporting, an automated tool that could screen patients for clinically significant MR (≥ moderate) would streamline the diagnostic/therapeutic pathways and ultimately improve patient outcomes. The authors aimed to develop and validate a fully automated machine learning (ML)-based echocardiography workflow for grading MR severity. ML algorithms were trained on echocardiograms from 2 observational cohorts and validated in patients from 2 additional independent studies. Multiparametric echocardiography core laboratory MR assessment served as ground truth. The machine was trained to measure 16 MR-related parameters. Multiple ML models were developed to find the optimal parameters and preferred ML model for MR severity grading. The preferred ML model used 9 parameters. Image analysis was feasible in 99.3% of cases and took 80 ± 5 seconds per case. The accuracy for grading MR severity (none to severe) was 0.80, and for significant (moderate or severe) vs nonsignificant MR was 0.97 with a sensitivity of 0.96 and specificity of 0.98. The model performed similarly in cases of eccentric and central MR. Patients graded as having severe MR had higher 1-year mortality (adjusted HR: 5.20 [95%CI: 1.24-21.9]; P = 0.025 compared with mild). An automated multiparametric ML model for grading MR severity is feasible, fast, highly accurate, and predicts 1-year mortality. Its implementation in clinical practice could improve patient care by facilitating referral to specialized clinics and access to evidence-based therapies while improving quality and efficiency in the echocardiography laboratory.

Neural Network Models for Ionospheric Electron Density Prediction at a Fixed Altitude Using Neural Architecture Search

AbstractSpecification and forecast of ionospheric parameters, such as ionospheric electron density (Ne), have been an important topic in space weather and ionospheric research. Neural networks (NNs) emerge as a powerful modeling tool for Ne prediction. However, heavy manual adjustments are time consuming to determine the optimal NN structures. In this work, we propose to use neural architecture search (NAS), an automatic machine learning method, to mitigate this problem. NAS aims to find the optimal network structure through the alternate optimization of the hyperparameters and the corresponding network parameters within a pre‐defined hyperparameter search space. A total of 16‐year data from Millstone Hill incoherent scatter radar (ISR) are used for the NN models. One single‐layer NN (SLNN) model and one deep NN (DNN) model are both trained with NAS, namely SLNN‐NAS and DNN‐NAS, for Ne prediction and compared with their manually tuned counterparts (SLNN and DNN) based on previous studies. Our results show that SLNN‐NAS and DNN‐NAS outperformed SLNN and DNN, respectively. These NN predictions of Ne daily variation patterns reveal a 27‐day mid‐latitude topside Ne variation, which cannot be reasonably represented by traditional empirical models developed using monthly averages. DNN‐NAS yields the best prediction accuracy measured by quantitative metrics and rankings of daily pattern prediction, especially with an improvement in mean absolute error more than 10% compared to the SLNN model. The limited improvement of NAS is likely due to the network complexity and the limitation of fully connected NN without the time histories of input parameters.

Redefining Neural Architecture Search of Heterogeneous Multinetwork Models by Characterizing Variation Operators and Model Components.

With neural architecture search (NAS) methods gaining ground on manually designed deep neural networks-even more rapidly as model sophistication escalates-the research trend is shifting toward arranging different and often increasingly complex NAS spaces. In this conjuncture, delineating algorithms which can efficiently explore these search spaces can result in a significant improvement over currently used methods, which, in general, randomly select the structural variation operator, hoping for a performance gain. In this article, we investigate the effect of different variation operators in a complex domain, that of multinetwork heterogeneous neural models. These models have an extensive and complex search space of structures as they require multiple subnetworks within the general model in order to answer different output types. From that investigation, we extract a set of general guidelines whose application is not limited to that particular type of model and are useful to determine the direction in which an architecture optimization method could find the largest improvement. To deduce the set of guidelines, we characterize both the variation operators, according to their effect on the complexity and performance of the model; and the models, relying on diverse metrics which estimate the quality of the different parts composing it.

Distributed quantum architecture search

Distributed quantum architecture search

Research on the artistic form and image expression effect of public sculpture based on intelligent and parametric design

With the modernization of cities, public sculptures are constantly being designed and constructed. The artistic form and image expression effect of sculpture based on intelligent and parametric design needs to be designed and developed to guide and assist the construction of sculpture. This paper applies the NAS architecture search method to explore the field of image expression effect models. Through the end-to-end search of the experiment designed in this paper, the separable convolution lightweight design is used, and the new model AestheticNet is used to predict the image form effect score distribution. Secondly, this paper proposes optimization strategies combining image expression effect theory and convolutional neural network, including improvement of Loss function self-weighted Loss, two-dimensional Attention mechanism – introduction of CBAM, and adaptive pooling layer. Optimization of several aspects, such as adaptive input. Finally, the validation set is compared with other existing image-morphological effect model algorithms, which proves the effectiveness of the customized search scheme. It demonstrates the efficacy of the AestheticNet model compared to other algorithms by validating its prediction of public sculpture image form effect ratings. The artistic form using intelligent and parametric design methodologies may improve. Image expression of sculptures may be enhanced by applying the image form effect model, which should be pervasive. We can use it to intelligently and parametrically guide the design and manufacture of sculptures.

Augmenting machine learning of Grad–Shafranov equilibrium reconstruction with Green's functions

This work presents a method for predicting plasma equilibria in tokamak fusion experiments and reactors. The approach involves representing the plasma current as a linear combination of basis functions using principal component analysis of plasma toroidal current densities (Jt) from the EFIT-AI equilibrium database. Then utilizing EFIT's Green's function tables, basis functions are created for the poloidal flux (ψ) and diagnostics generated from the toroidal current (Jt). Similar to the idea of a physics-informed neural network (NN), this physically enforces consistency between ψ, Jt, and the synthetic diagnostics. First, the predictive capability of a least squares technique to minimize the error on the synthetic diagnostics is employed. The results show that the method achieves high accuracy in predicting ψ and moderate accuracy in predicting Jt with median R2 = 0.9993 and R2 = 0.978, respectively. A comprehensive NN using a network architecture search is also employed to predict the coefficients of the basis functions. The NN demonstrates significantly better performance compared to the least squares method with median R2 = 0.9997 and 0.9916 for Jt and ψ, respectively. The robustness of the method is evaluated by handling missing or incorrect data through the least squares filling of missing data, which shows that the NN prediction remains strong even with a reduced number of diagnostics. Additionally, the method is tested on plasmas outside of the training range showing reasonable results.

Short-term load forecasting of mixed industrial users in distribution networks using DARTS-LSTM method

Abstract Load forecasting has perennially stood as a fundamental pillar within the power industry, with precision in prognostication constituting a paramount necessity for critical functions including grid management, optimization of energy storage, and response to demand fluctuations. In the present study, we introduce a comprehensive global modeling prediction framework aimed at forecasting large-scale power demand within time series data, alongside the development of a novel short-term power load forecasting model. By integrating all time series data pertaining to the same regression task and employing a unified univariate prediction function across the entire time series dataset, we efficaciously forecast large-scale heterogeneous loads aggregated at various levels within distribution networks. The proposed framework leverages the cutting-edge DARTS (Differentiable Architecture Search) deep learning framework for implementation. Through comprehensive experimentation on real-world data, our model framework demonstrates superior performance in terms of overall prediction accuracy, scalability, and robustness in handling missing and non-stationary time series data.

Combining Compressed Sensing and Neural Architecture Search for Sensor-Near Vibration Diagnostics

Combining Compressed Sensing and Neural Architecture Search for Sensor-Near Vibration Diagnostics

Applications and issues of artificial intelligence in the financial sector

This paper investigates the application of artificial intelligence in the financial sector, analyzing the existing technical, ethical, and legal issues, and proposing corresponding solutions. The research background highlights the widespread use of AI technology in the financial industry and the efficiency and cost benefits it brings. The research focuses on challenges related to data quality, feature engineering, model complexity, real-time capability, computational resources, and data privacy protection. The research method includes literature review and case analysis, revealing the applications of AI technology in stock prediction, risk management, trading strategy optimization, and customer service. The results indicate that effective data cleaning, automated feature engineering, model simplification and regularization techniques, the use of interpretability tools like LIME and SHAP, and the introduction of fairness evaluation standards can significantly enhance AI model performance and transparency. The conclusion points out that these measures can not only solve the current technical and ethical issues of AI in the financial sector but also promote the widespread application and standardization of AI technology in other fields.

Predicting the compressive strength of engineered geopolymer composites using automated machine learning

Engineered Geopolymer Composites (EGC) offer a sustainable and high-performance alternative to traditional concrete and Engineered Cementitious Composites, boasting reduced environmental impact and enhanced mechanical properties. However, optimising EGC mix design for specific applications requires an accurate prediction of its compressive strength. This study investigates the application of Automated Machine Learning using the PyCaret library to develop reliable predictive models for EGC compressive strength. A comprehensive experimental program was conducted, testing 132 EGC specimens with varying mix design parameters, including binder ratio, silica fume content, activator ratio, water content, superplasticizer dosage, and curing method. The collected data was used to train and evaluate twenty different machine learning models. Model performance was assessed using various metrics. The top six models were shortlisted and optimised using Random Search algorithm. The models were assessed through a detailed analysis of their residual plots and learning curves. Additionally, Feature importance and SHAP analysis were employed to understand the influence of each input parameter on the predicted compressive strength. The results demonstrate the effectiveness of AML in accurately predicting EGC compressive strength, with the Gradient Boosting Regressor and CatBoost Regressor models exhibiting superior performance, achieving Mean Absolute Error (MAE) below 1.2 MPa and R² exceeding 0.96. The study highlights the potential of AML as a valuable tool for optimising EGC mix design and promoting the wider adoption of this sustainable construction material.

AutoCTS++: zero-shot joint neural architecture and hyperparameter search for correlated time series forecasting

AutoCTS++: zero-shot joint neural architecture and hyperparameter search for correlated time series forecasting

Подход к настройке CatBoost для продвинутого обнаружения атак DoS и DDoS в сетевом трафике

In the ever-evolving landscape of network security, the sophistication of cyber-attacks, especially Denial of Service (DoS) and Distributed Denial of Service (DDoS) attacks, poses a formidable challenge to intrusion detection systems. Recognizing the longstanding application of CatBoost in various domains, this study explores its novel optimization for network intrusion detection, a critical area in need of advanced solutions. Leveraging the strengths of CatBoost in handling categorical data and imbalanced datasets, we meticulously adapt the classifier to meet the complex demands of distinguishing between DoS, DDoS, and benign traffic within the comprehensive CICIDS2017 and CSE-CIC-IDS2018 datasets. This research is an attempt to refine the learning efficiency and detection capabilities of CatBoost through the implementation of advanced feature selection and data preparation, contributing to the field by improving detection accuracy within real-time intrusion detection systems. The results show a notable improvement in performance, underscoring the classifier's role in advancing cybersecurity measures. Furthermore, the study paves the way for future exploration into adversarial machine learning and automated feature engineering, fortifying the resilience and adaptability of intrusion detection systems against the backdrop of a rapidly changing cyber threat landscape. These efforts provide solid approaches to address the current challenges in network security, signaling a move towards more refined and dependable intrusion detection methods.

Machine Learning in Medical Triage: A Predictive Model for Emergency Department Disposition

The study explores the application of automated machine learning (AutoML) using the MIMIC-IV-ED database to enhance decision-making in emergency department (ED) triage. We developed a predictive model that utilizes triage data to forecast hospital admissions, aiming to support medical staff by providing an advanced decision-support system. The model, powered by H2O.ai’s AutoML platform, was trained on approximately 280,000 preprocessed records from the Beth Israel Deaconess Medical Center collected between 2011 and 2019. The selected Gradient Boosting Machine (GBM) model demonstrated an AUC ROC of 0.8256, indicating its efficacy in predicting patient dispositions. Key variables such as acuity and waiting hours were identified as significant predictors, emphasizing the model’s capability to integrate critical triage metrics into its predictions. However, challenges related to the complexity and heterogeneity of medical data, privacy concerns, and the need for model interpretability were addressed through the incorporation of Explainable AI (XAI) techniques. These techniques ensure the transparency of the predictive processes, fostering trust and facilitating ethical AI use in clinical settings. Future work will focus on external validation and expanding the model to include a broader array of variables from diverse healthcare environments, enhancing the model’s utility and applicability in global emergency care contexts.

SalNAS: Efficient Saliency-prediction Neural Architecture Search with self-knowledge distillation

Recent advancements in deep convolutional neural networks have significantly improved the performance of saliency prediction. However, the manual configuration of the neural network architectures requires domain knowledge expertise and can still be time-consuming and error-prone. To solve this, we propose a new Neural Architecture Search (NAS) framework for saliency prediction with two contributions. Firstly, a supernet for saliency prediction is built with a weight-sharing network containing all candidate architectures, by integrating a dynamic convolution into the encoder–decoder in the supernet, termed SalNAS. Secondly, despite the fact that SalNAS is highly efficient (20.98 million parameters), it can suffer from the lack of generalization. To solve this, we propose a self-knowledge distillation approach, termed Self-KD, that trains the student SalNAS with the weighted average information between the ground truth and the prediction from the teacher model. The teacher model, while sharing the same architecture, contains the best-performing weights chosen by cross-validation. Self-KD can generalize well without the need to compute the gradient in the teacher model, enabling an efficient training system. By utilizing Self-KD, SalNAS outperforms other state-of-the-art saliency prediction models in most evaluation rubrics across seven benchmark datasets while being a lightweight model. The code will be available at https://github.com/chakkritte/SalNAS.

Image-based quantification of pool boiling heat flux on varied heating surfaces: Enhancing prediction performance with automated machine learning

The present study applied image-trained Convolutional Neural Networks (CNN) to quantify the heat flux dissipated in a pool boiling experiment. In total, four heating surfaces were used to generate over 200 thousand nucleate and film boiling images in two visualization modes: direct and indirect. In the former, the flow images included the heating surface; in the latter, the image was cropped and the heating surface was omitted. Prior to training and testing the CNN, the images were preprocessed, by grayscaling, downscaling, size uniformization and standardization. The main goals were to assess the CNN’s capability to generalize to multiple operating conditions, and to quantify the performance of a CNN architecture optimized with Automated Machine Learning (AutoML) in comparison with a reference architecture. The results suggest that multi-dataset training is required to improve the CNN generalization. In other words, the CNN must be trained, even if partially, with images of the specific heating surface for which it is trying to infer the heat flux. However, the results indicate a significant performance drop for the results of multi-surface trained CNNs when compared with single-surface CNNs, suggesting limited generalization capability. Furthermore, the results obtained showed that AutoML was capable of increasing the performance of CNN models when compared with parametrically determined architectures. Also, optimized architectures tend to present a larger number of convolutional layers associated with dense blocks and, at the same time, a reduced number of trainable variables.

Multi-objective meta-learning

Meta-learning has arisen as a powerful tool for many machine learning problems. With multiple factors to be considered when designing learning models for real-world applications, meta-learning with multiple objectives has attracted much attention recently. However, existing works either linearly combine multiple objectives into one objective or adopt evolutionary algorithms to handle it, where the former approach needs to pay high computational cost to tune the combination coefficients while the latter approach is computationally heavy and incapable to be integrated into gradient-based optimization. To alleviate those limitations, in this paper, we aim to propose a generic gradient-based Multi-Objective Meta-Learning (MOML) framework with applications in many machine learning problems. Specifically, the MOML framework formulates the objective function of meta-learning with multiple objectives as a Multi-Objective Bi-Level optimization Problem (MOBLP) where the upper-level subproblem is to solve several possibly conflicting objectives for the meta-learner. Different from those existing works, in this paper, we propose a gradient-based algorithm to solve the MOBLP. Specifically, we devise the first gradient-based optimization algorithm by alternately solving the lower-level and upper-level subproblems via the gradient descent method and the gradient-based multi-objective optimization method, respectively. Theoretically, we prove the convergence property and provide a non-asymptotic analysis of the proposed gradient-based optimization algorithm. Empirically, extensive experiments justify our theoretical results and demonstrate the superiority of the proposed MOML framework for different learning problems, including few-shot learning, domain adaptation, multi-task learning, neural architecture search, and reinforcement learning. The source code of MOML is available at https://github.com/Baijiong-Lin/MOML.

Automated machine learning for fabric quality prediction: a comparative analysis.

The enhancement of fabric quality prediction in the textile manufacturing sector is achieved by utilizing information derived from sensors within the Internet of Things (IoT) and Enterprise Resource Planning (ERP) systems linked to sensors embedded in textile machinery. The integration of Industry 4.0 concepts is instrumental in harnessing IoT sensor data, which, in turn, leads to improvements in productivity and reduced lead times in textile manufacturing processes. This study addresses the issue of imbalanced data pertaining to fabric quality within the textile manufacturing industry. It encompasses an evaluation of seven open-source automated machine learning (AutoML) technologies, namely FLAML (Fast Lightweight AutoML), AutoViML (Automatically Build Variant Interpretable ML models), EvalML (Evaluation Machine Learning), AutoGluon, H2OAutoML, PyCaret, and TPOT (Tree-based Pipeline Optimization Tool). The most suitable solutions are chosen for certain circumstances by employing an innovative approach that finds a compromise among computational efficiency and forecast accuracy. The results reveal that EvalML emerges as the top-performing AutoML model for a predetermined objective function, particularly excelling in terms of mean absolute error (MAE). On the other hand, even with longer inference periods, AutoGluon performs better than other methods in measures like mean absolute percentage error (MAPE), root mean squared error (RMSE), and r-squared. Additionally, the study explores the feature importance rankings provided by each AutoML model, shedding light on the attributes that significantly influence predictive outcomes. Notably, sin/cos encoding is found to be particularly effective in characterizing categorical variables with a large number of unique values. This study includes useful information about the application of AutoML in the textile industry and provides a roadmap for employing Industry 4.0 technologies to enhance fabric quality prediction. The research highlights the importance of striking a balance between predictive accuracy and computational efficiency, emphasizes the significance of feature importance for model interpretability, and lays the groundwork for future investigations in this field.

Auto-Spikformer: Spikformer architecture search.

The integration of self-attention mechanisms into Spiking Neural Networks (SNNs) has garnered considerable interest in the realm of advanced deep learning, primarily due to their biological properties. Recent advancements in SNN architecture, such as Spikformer, have demonstrated promising outcomes. However, we observe that Spikformer may exhibit excessive energy consumption, potentially attributable to redundant channels and blocks. To mitigate this issue, we propose a one-shot Spiking Transformer Architecture Search method, namely Auto-Spikformer. Auto-Spikformer extends the search space to include both transformer architecture and SNN inner parameters. We train and search the supernet based on weight entanglement, evolutionary search, and the proposed Discrete Spiking Parameters Search (DSPS) methods. Benefiting from these methods, the performance of subnets with weights inherited from the supernet without even retraining is comparable to the original Spikformer. Moreover, we propose a new fitness function aiming to find a Pareto optimal combination balancing energy consumption and accuracy. Our experimental results demonstrate the effectiveness of Auto-Spikformer, which outperforms the original Spikformer and most CNN or ViT models with even fewer parameters and lower energy consumption.