Neural Network Hyperparameters Research Articles

Artificial neural network hyperparameters optimization for predicting the thermal conductivity of MXene/graphene nanofluids

BackgroundThe critical role of thermal conductivity (TC) as a significant thermo-physical property in MXene/graphene-based nanofluids for photovoltaic/thermal systems has motivated recent research into developing precision predictive models. The multilayer perceptron neural network (MLPNN) has emerged as an eminent AI algorithm for this task. MethodsThis study employs Bayesian optimization, random search (RS), and grid search (GS) to fine-tune MLPNN hyperparameters—hidden layers, neurons, activation functions, standardization, and regularization—to elevate TC modeling efficiency. The proposed methodology unfolds in sequential phases: data analysis, data pre-processing, and introduction of MLPNN, GS, RS, Bayesian approach, and their integration algorithm. The next phase entails developing predictive models and presenting optimal cases. Lastly, the final models undergo statistical evaluation and graphical comparison for a thorough analysis. FindingsResults manifest that the GS-MLPNN model excels, achieving the lowest testing data error (MAPE = 0.5261%) and high conformity with empirical data (R = 0.99941). Meanwhile, the RS method adjusts hyperparameters with negligible precision loss (MAPE = 0.6046%, R = 0.99887). Contrarily, Bayesian optimization lags, increasing errors (MAPE = 3.1981%) and lower correlation (R = 0.98099), suggesting its relative inefficacy for this specific application. The optimized models provide efficient predictions, significantly reducing the financial/computing costs associated with experimental/numerical analysis.

NPP accident prevention: Integrated neural network for coupled multivariate time series prediction based on PSO and its application under uncertainty analysis for NPP data

Due to the requirement of a cleaner, more sustainable form of energy production and the rapid development of DT, nuclear energy needs to be digital transformed. This paper aims at the time series prediction module of DT in NPP. Despite the widespread application of time series forecasting in various domains, the inherent uncertainties within the data and the selection of neural network hyperparameters pose a formidable challenge to accurate predictions. This study proposes a coupled multivariate prediction method and conducts a comparative analysis of three popular time series forecasting models: LSTM, CNN-LSTM, and Transformer. PSO hyperparameter optimization is also employed, accompanied by multiple error calculation metrics for model evaluation. It is noteworthy that the experimental data employed in this study are derived from the real operation process of a Gen II + reactor in the Chinese mainland, which is in the off-site power loss accident condition. Simultaneously, adopting a smaller training dataset proportion contributes to striking a balance among resource utilization, model performance, and experimental efficiency in the research. Experimental results unequivocally demonstrate that among these models, Transformer excels in multi-input multi-output time series forecasting. Moreover, uncertainty analysis using Bayesian generates a forecast band, proving the robustness of Transformer.

Using swarm intelligence to optimize neural network hyperparameters: comparative analysis on MNIST and CIFAR-10

Using swarm intelligence to optimize neural network hyperparameters: comparative analysis on MNIST and CIFAR-10

ML-driven segmentation of microvascular features during histological examination of tissue-engineered vascular grafts.

The development of next-generation tissue-engineered medical devices such as tissue-engineered vascular grafts (TEVGs) is a leading trend in translational medicine. Microscopic examination is an indispensable part of animal experimentation, and histopathological analysis of regenerated tissue is crucial for assessing the outcomes of implanted medical devices. However, the objective quantification of regenerated tissues can be challenging due to their unusual and complex architecture. To address these challenges, research and development of advanced ML-driven tools for performing adequate histological analysis appears to be an extremely promising direction. We compiled a dataset of 104 representative whole slide images (WSIs) of TEVGs which were collected after a 6-month implantation into the sheep carotid artery. The histological examination aimed to analyze the patterns of vascular tissue regeneration in TEVGs in situ. Having performed an automated slicing of these WSIs by the Entropy Masker algorithm, we filtered and then manually annotated 1,401 patches to identify 9 histological features: arteriole lumen, arteriole media, arteriole adventitia, venule lumen, venule wall, capillary lumen, capillary wall, immune cells, and nerve trunks. To segment and quantify these features, we rigorously tuned and evaluated the performance of six deep learning models (U-Net, LinkNet, FPN, PSPNet, DeepLabV3, and MA-Net). After rigorous hyperparameter optimization, all six deep learning models achieved mean Dice Similarity Coefficients (DSC) exceeding 0.823. Notably, FPN and PSPNet exhibited the fastest convergence rates. MA-Net stood out with the highest mean DSC of 0.875, demonstrating superior performance in arteriole segmentation. DeepLabV3 performed well in segmenting venous and capillary structures, while FPN exhibited proficiency in identifying immune cells and nerve trunks. An ensemble of these three models attained an average DSC of 0.889, surpassing their individual performances. This study showcases the potential of ML-driven segmentation in the analysis of histological images of tissue-engineered vascular grafts. Through the creation of a unique dataset and the optimization of deep neural network hyperparameters, we developed and validated an ensemble model, establishing an effective tool for detecting key histological features essential for understanding vascular tissue regeneration. These advances herald a significant improvement in ML-assisted workflows for tissue engineering research and development.

Neural Architecture Search Using Covariance Matrix Adaptation Evolution Strategy.

Evolution-based neural architecture search methods have shown promising results, but they require high computational resources because these methods involve training each candidate architecture from scratch and then evaluating its fitness, which results in long search time. Covariance Matrix Adaptation Evolution Strategy (CMA-ES) has shown promising results in tuning hyperparameters of neural networks but has not been used for neural architecture search. In this work, we propose a framework called CMANAS which applies the faster convergence property of CMA-ES to the deep neural architecture search problem. Instead of training each individual architecture seperately, we used the accuracy of a trained one shot model (OSM) on the validation data as a prediction of the fitness of the architecture, resulting in reduced search time. We also used an architecture-fitness table (AF table) for keeping a record of the already evaluated architecture, thus further reducing the search time. The architectures are modeled using a normal distribution, which is updated using CMA-ES based on the fitness of the sampled population. Experimentally, CMANAS achieves better results than previous evolution-based methods while reducing the search time significantly. The effectiveness of CMANAS is shown on two different search spaces using four datasets: CIFAR-10, CIFAR-100, ImageNet, and ImageNet16-120. All the results show that CMANAS is a viable alternative to previous evolution-based methods and extends the application of CMA-ES to the deep neural architecture search field.

The unreasonable effectiveness of early discarding after one epoch in neural network hyperparameter optimization

To reach high performance with deep learning, hyperparameter optimization (HPO) is essential. This process is usually time-consuming due to costly evaluations of neural networks. Early discarding techniques limit the resources granted to unpromising candidates by observing the empirical learning curves and canceling neural network training as soon as the lack of competitiveness of a candidate becomes evident. Despite two decades of research, little is understood about the trade-off between the aggressiveness of discarding and the loss of predictive performance. Our paper studies this trade-off for several commonly used discarding techniques such as successive halving and learning curve extrapolation. Our surprising finding is that these commonly used techniques offer minimal to no added value compared to the simple strategy of discarding after a constant number of epochs of training. The chosen number of epochs mostly depends on the available compute budget. We call this approach i-Epoch (i being the constant number of epochs with which neural networks are trained) and suggest to assess the quality of early discarding techniques by comparing how their Pareto-Front (in consumed training epochs and predictive performance) complement the Pareto-Front of i-Epoch.

Predicting Diffusion Coefficients in Nafion Membranes during the Soaking Process Using a Machine Learning Approach.

Nafion, a versatile polymer used in electrochemistry and membrane technologies, exhibits complex behaviors in saline environments. This study explores Nafion membrane's IR spectra during soaking and subsequent drying processes in salt solutions at various concentrations. Utilizing the principles of Fick's second law, diffusion coefficients for these processes are derived via exponential approximation. By harnessing machine learning (ML) techniques, including the optimization of neural network hyperparameters via a genetic algorithm (GA) and leveraging various regressors, we effectively pinpointed the optimal model for predicting diffusion coefficients. Notably, for the prediction of soaking coefficients, our model is composed of layers with 64, 64, 32, and 16 neurons, employing ReLU, ELU, sigmoid, and ELU activation functions, respectively. Conversely, for drying coefficients, our model features two hidden layers with 16 and 12 neurons, utilizing sigmoid and ELU activation functions, respectively.

I Choose You: Automated Hyperparameter Tuning for Deep Learning-Based Side-Channel Analysis

Today, the deep learning-based side-channel analysis represents a widely researched topic, with numerous results indicating the advantages of such an approach. Indeed, breaking protected implementations while not requiring complex feature selection made deep learning a preferred option for profiling side-channel analysis. Still, this does not mean it is trivial to mount a successful deep learning-based side-channel analysis. One of the biggest challenges is to find optimal hyperparameters for neural networks resulting in powerful side-channel attacks. This work proposes an automated way for deep learning hyperparameter tuning based on Bayesian optimization. We build a custom framework denoted AutoSCA supporting machine learning and side-channel metrics. Our experimental analysis shows that our framework performs well regardless of the dataset, leakage model, or neural network type. We find several neural network architectures outperforming state-of-the-art attacks. Finally, while not considered a powerful option, we observe that neural networks obtained via random search can perform well, indicating that the publicly available datasets are relatively easy to break.

Research on optimization of key thermal parameters of the secondary loop of PWR based on improved BP neural network

The simulation and optimization time of traditional secondary loop thermal parameters is relatively long, and the computational cost is generally high. To solve this problem, we construct an accurate model of the full-power steady-state operation of the secondary loop of the PWR, and use meta-learning combined with Bayesian optimization to realize the adaptive optimization of BP neural network hyperparameters. The effectiveness of the method is verified by designing comparative experiments by taking thermal efficiency as the optimization goal and high-pressure heater end difference and other parameters as the optimization variables, and using the improved BP neural network to optimize the accurate model. The results show that the improved secondary loop thermal parameter prediction and optimization framework can quickly and accurately predict the thermal efficiency and optimize the thermal parameters, and has good generalization ability, which lays a foundation for improving the thermal efficiency of nuclear power plants.

Accurate detection and intelligent classification of solar cells defects based on photoluminescence images: A novel study on the optimized YOLOv5 model

In the production process of solar cells, inevitable faults such as cracks, dirt, dark spots, and scratches may occur, which could potentially impact the lifespan and power generation efficiency of solar cells. Addressing this issue, this paper combines neural networks with photoluminescence detection technology and proposes a novel neural network model for the classification and grading of defects in solar cells. Firstly, the YOLOv5 model is optimized and adjusted for algorithm and network structure. The optimization process is divided into three parts: global optimization of the network structure, optimization of the neck network structure, and optimization of the head structure, each addressing specific issues in recognition, detection, and classification. The impact of the optimized network model on recognition and detection speed is analyzed, and solutions are proposed to address any observed effects. Additionally, an iterative update of neural network hyperparameter combinations is performed for solar cell defect identification. Finally, using the ultimately optimized model structure in conjunction with the optimal hyperparameter combination, comparative experiments are conducted on neural networks for different target identification using the photoluminescence characteristics dataset of solar cells. The recognition improvement of the optimized model and its differences from other models are analyzed.

OPG-based dental age estimation using a data-technical exploration of deep learning techniques.

Dental age estimation, a cornerstone in forensic age assessment, has been extensively tried and tested, yet manual methods are impeded by tedium and interobserver variability. Automated approaches using deep transfer learning encounter challenges like data scarcity, suboptimal training, and fine-tuning complexities, necessitating robust training methods. This study explores the impact of convolutional neural network hyperparameters, model complexity, training batch size, and sample quantity on age estimation. EfficientNet-B4, DenseNet-201, and MobileNet V3 models underwent cross-validation on a dataset of 3896 orthopantomograms (OPGs) with batch sizes escalating from 10 to 160 in a doubling progression, as well as random subsets of this training dataset. Results demonstrate the EfficientNet-B4 model, trained on the complete dataset with a batch size of 160, as the top performer with a mean absolute error of 0.562 years on the test set, notably surpassing the MAE of 1.01 at a batch size of 10. Increasing batch size consistently improved performance for EfficientNet-B4 and DenseNet-201, whereas MobileNet V3 performance peaked at batch size 40. Similar trends emerged in training with reduced sample sizes, though they were outperformed by the complete models. This underscores the critical role of hyperparameter optimization in adopting deep learning for age estimation from complete OPGs. The findings not only highlight the nuanced interplay of hyperparameters and performance but also underscore the potential for accurate age estimation models through optimization. This study contributes to advancing the application of deep learning in forensic age estimation, emphasizing the significance of tailored training methodologies for optimal outcomes.

An LSTM-stacked autoencoder multisource response prediction and constraint optimization for scaled expansion tubes

Deep learning is attracting increasing attention due to its excellent predictive power and its ability to be applied in traditional research areas. In this paper, we propose a multisource response prediction network architecture based on a long short-term memory (LSTM)-stacked autoencoder to predict key crashworthiness indicators and complete curve reconstruction. Taking an expansion tube as an example using an equivalent scaling research method, a scaled expandable tubular (SET) finite element model was established and verified using a quasistatic compression test and a full-size coupler and buffer system experiment. A design of experiments (DOE) approach was used to obtain a dataset for training the prediction network. Neural network hyperparameters are critical to network prediction accuracy, and after comparison, the multisource response prediction network architecture showed good computational efficiency and satisfactory prediction accuracy when appropriate hyperparameters were selected. Subsequently, multiobjective constraint optimization was performed using the nondominated sorting genetic algorithm-II (NSGA-II) based on a prediction network architecture, which greatly improved the energy-absorption structure optimization accuracy. The results are expected to provide a research methodology for solving complex engineering problems by establishing a new framework for deep learning algorithms combined with optimization methods.

Boosting aquila optimizer by marine predators algorithm for combinatorial optimization

Abstract In this study, an improved version of aquila optimizer (AO) known as EHAOMPA has been developed by using the marine predators algorithm (MPA). MPA is a recent and well-behaved optimizer with a unique memory saving and fish aggregating devices mechanism. At the same time, it suffers from various defects such as inadequate global search, sluggish convergence, and stagnation of local optima. However, AO has contented robust global exploration capability, fast convergence speed, and high search efficiency. Thus, the proposed EHAOMPA aims to complement the shortcomings of AO and MPA while bringing new features. Specifically, the representative-based hunting technique is incorporated into the exploration stage to enhance population diversity. At the same time, random opposition-based learning is introduced into the exploitation stage to prevent the optimizer from sticking to local optima. This study tests the performance of EHAOMPA’s on 23 standard mathematical benchmark functions, 29 complex test functions from the CEC2017 test suite, six constrained industrial engineering design problems, and a convolutional neural network hyperparameter (CNN-hyperparameter) optimization for Corona Virus Disease 19 (COVID-19) computed tomography-image detection problem. EHAOMPA is compared with four existing optimization algorithm types, achieving the best performance on both numerical and practical issues. Compared with other methods, the test function results demonstrate that EHAOMPA exhibits a more potent global search capability, a higher convergence rate, increased accuracy, and an improved ability to avoid local optima. The excellent experimental results in practical problems indicate that the developed EHAOMPA has great potential in solving real-world optimization problems. The combination of multiple strategies can effectively improve the performance of the algorithm. The source code of the EHAOMPA is publicly available at https://github.com/WangShuang92/EHAOMPA.

Convolutional neural network hyperparameters for face emotion recognition using genetic algorithm

The development of artificial intelligence in facial emotion recognition (FER) is rapidly growing and has been widely applied in various fields. Deep learning (DL) techniques with evolutionary algorithms have become the preferred choice for solving various security, health, gaming, and other related problems. This research proposes the use of a genetic algorithm (GA) as the main method to optimize hyperparameters in the convolutional neural network (CNN) model for FER. The required computation time is approximately 37 hours 57 minutes 55 seconds, with generation 3 taking the longest time at around 16 hours 45 minutes 4 seconds. However, generation 3 achieved an accuracy of 76.11%, which is the highest compared to other generations. The results indicate that the more generations are involved, the higher the achievable accuracy. Furthermore, the proposed CNN-GA model in this study outperforms previous models that have been examined. Thus, this study makes a significant contribution to improving the understanding of using GAs to optimize the performance of CNN models for FER.

Asynchronous evolution of deep neural network architectures

Many evolutionary algorithms (EAs) take advantage of parallel evaluation of candidates. However, if evaluation times vary significantly, many worker nodes (i.e., compute clients) are idle much of the time, waiting for the next generation to be created. Evolutionary neural architecture search (ENAS), a class of EAs that optimizes the architecture and hyperparameters of deep neural networks, is particularly vulnerable to this issue. This paper proposes a generic asynchronous evaluation strategy (AES) that is then adapted to work with ENAS. AES increases throughput by maintaining a queue of up to K individuals ready to be sent to the workers for evaluation and proceeding to the next generation as soon as M<<K individuals have been evaluated. A suitable value for M is determined experimentally, balancing diversity and efficiency. To showcase the generality and power of AES, it was first evaluated in eight-line sorting network design (a single-population optimization task with limited evaluation-time variability), achieving an over two-fold speedup. Next, it was evaluated in 11-bit multiplexer design (a single-population discovery task with extended variability), where a 14-fold speedup was observed. It was then scaled up to ENAS for image captioning (a multi-population open-ended-optimization task), resulting in an over two-fold speedup. In all problems, a multifold performance improvement was observed, suggesting that AES is a promising method for parallelizing the evolution of complex systems with long and variable evaluation times, such as those in ENAS.

PREDICTION SYSTEM FOR THE INFLUENCE OF PHOSPHORUS ON WHEAT YIELD: OPTIMAL HYPERPARAMETER SELECTION

Modern studies in the application of machine learning in agrotechnology have showcased significant advancements in accurate crop yield forecasting and agricultural production optimization. This study aimed to select optimal hyperparameters for a neural network designed to forecast the impact of phosphorus on the yield of spring wheat. Tasks to achieve this goal included the selection of optimal neural network hyperparameters and deployment of the prediction model into forecasting system, ensuring accessibility and user convenience. Input dataset consisted of year, region, climatic indicators (temperature of the soil surface, precipitation, humidity), applied phosphorus, and the target variable is the yield of spring wheat. To select the optimal hyperparameters, GridSearchCV was utilized and integrated into the experiment, enabling the achievement of more precise and impactful forecasting outcomes. Experiments aimed at hyperparameter tuning identified an optimal network configuration (3 layers, 32 neurons, 300 epochs, 32 batches). Results from training showcased that this network configuration achieved the best performance, demonstrating minimal mean squared error (MSE). These results introduced a novel scientific approach in training data using this neural network model for forecasting the yield of cereal crops. Developing the prediction system's interface relied on the Streamlit, allowing the creation of an intuitive and user-friendly interface. The visualization derived from this model empowers users to evaluate crop yield and receive recommendations for optimal phosphorus application to enhance spring wheat yield. These results highlighted practical significance of this prediction system, based on neural network utilizing optimal hyperparameters that are ready for practical implementation by system developers.

Automatic epileptic seizure detection based on EEG using a moth-flame optimization of one-dimensional convolutional neural networks.

Frequent epileptic seizures can cause irreversible damage to the brains of patients. A potential therapeutic approach is to detect epileptic seizures early and provide artificial intervention to the patient. Currently, extracting electroencephalogram (EEG) features to detect epileptic seizures often requires tedious methods or the repeated adjustment of neural network hyperparameters, which can be time- consuming and demanding for researchers. This study proposes an automatic detection model for an EEG based on moth-flame optimization (MFO) optimized one-dimensional convolutional neural networks (1D-CNN). First, according to the characteristics and need for early epileptic seizure detection, a data augmentation method for dividing an EEG into small samples is proposed. Second, the hyperparameters are tuned based on MFO and trained for an EEG. Finally, the softmax classifier is used to output EEG classification from a small-sample and single channel. The proposed model is evaluated with the Bonn EEG dataset, which verifies the feasibility of EEG classification problems that involve up to five classes, including healthy, preictal, and ictal EEG from various brain regions and individuals. Compared with existing advanced optimization algorithms, such as particle swarm optimization, genetic algorithm, and grey wolf optimizer, the superiority of the proposed model is further verified. The proposed model can be implemented into an automatic epileptic seizure detection system to detect seizures in clinical applications.

Gradient-enhanced physics-informed neural networks based on transfer learning for inverse problems of the variable coefficient differential equations

We propose gradient-enhanced PINNs based on transfer learning (TL-gPINNs) for inverse problems of the function coefficient discovery in order to overcome deficiency of the discrete characterization of the PDE loss in neural networks and improve accuracy of function feature description, which offers a new angle of view for gPINNs. The TL-gPINN algorithm is applied to infer the unknown variable coefficients of various forms (the polynomial, trigonometric function, hyperbolic function and fractional polynomial) and multiple variable coefficients simultaneously with abundant soliton solutions for the well-known variable coefficient nonlinear Schrödinger equation. Compared with the PINN and gPINN, TL-gPINN yields considerable improvement in accuracy. Moreover, our method leverages the advantage of the transfer learning technique, which can help to mitigate the problem of inefficiency caused by extra loss terms of the gradient. Numerical results fully demonstrate the effectiveness of the TL-gPINN method in significant accuracy enhancement, and it also outperforms gPINN in efficiency even when the training data was corrupted with different levels of noise or hyper-parameters of neural networks are arbitrarily changed.

Machine Learning for Nonorographic Gravity Waves in a Climate Model

Abstract There is growing use of machine learning algorithms to replicate subgrid parameterization schemes in global climate models. Parameterizations rely on approximations; thus, there is potential for machine learning to aid improvements. In this study, a neural network is used to mimic the behavior of the nonorographic gravity wave scheme used in the Met Office climate model, important for stratospheric climate and variability. The neural network is found to require only two of the six inputs used by the parameterization scheme, suggesting the potential for greater efficiency in this scheme. Use of a one-dimensional mechanistic model is advocated, allowing neural network hyperparameters to be chosen based on emergent features of the coupled system with minimal computational cost, and providing a testbed prior to coupling to a climate model. A climate model simulation, using the neural network in place of the existing parameterization scheme, is found to accurately generate a quasi-biennial oscillation of the tropical stratospheric winds, and correctly simulate the nonorographic gravity wave variability associated with El Niño–Southern Oscillation and stratospheric polar vortex variability. These internal sources of variability are essential for providing seasonal forecast skill, and the gravity wave forcing associated with them is reproduced without explicit training for these patterns. Significance Statement Climate simulations are required for providing advice to government, industry, and society regarding the expected climate on time scales of months to decades. Machine learning has the potential to improve the representation of some sources of variability in climate models that are too small to be directly simulated by the model. This study demonstrates that a neural network can simulate the variability due to atmospheric gravity waves that is associated with El Niño–Southern Oscillation and with the tropical and polar regions of the stratosphere. These details are important for a model to produce more accurate predictions of regional climate.

Electrical power output prediction of combined cycle power plants using a recurrent neural network optimized by waterwheel plant algorithm

It is difficult to analyze and anticipate the power output of Combined Cycle Power Plants (CCPPs) when considering operational thermal variables such as ambient pressure, vacuum, relative humidity, and temperature. Our data visualization study shows strong non-linearity in the experimental data. We observe that CCPP energy production increases linearly with temperature but not pressure. We offer the Waterwheel Plant Algorithm (WWPA), a unique metaheuristic optimization method, to fine-tune Recurrent Neural Network hyperparameters to improve prediction accuracy. A robust mathematical model for energy production prediction is built and validated using anticipated and experimental data residuals. The residuals’ uniformity above and below the regression line suggests acceptable prediction errors. Our mathematical model has an R-squared value of 0.935 and 0.999 during training and testing, demonstrating its outstanding predictive accuracy. This research provides an accurate way to forecast CCPP energy output, which could improve operational efficiency and resource utilization in these power plants.