Neural Network Optimization Method Research Articles

An adaptive multi-objective optimization method for structure-connection-performance design of commercial vehicle cab

A variable screening-approximate model-adaptive multi-optimization algorithm integrated method is proposed to solve the high-dimensional nonlinear problems for lightweight optimization of cabs. Firstly, a full-parametric body-in-white model of the cab is established using implicit parameterization technology, and the simulation analysis of the bending and torsional stiffness, natural modal characteristics, and passive safety are established. Then, considering the structure and connection process parameters, the design variables are screened based on a gray relational analysis-TOPSIS (GRA-TOPSIS). Finally, integrated multi-objective optimization of cab structure-connection-performance is carried out by an adaptive RBF neural network optimization method, and the optimal solution is determined by the GRA-TOPSIS method. Compared with the initial model, the mass of cab is reduced by 3.28% after optimization design. It provides guidance for the lightweight design of body.

MFFGD: An adaptive Caputo fractional-order gradient algorithm for DNN

As a primary optimization method for neural networks, gradient descent algorithm has received significant attention in the recent development of deep neural networks. However, current gradient descent algorithms still suffer from drawbacks such as an excess of hyperparameters, getting stuck in local optima, and poor generalization. This paper introduces a novel Caputo fractional-order gradient descent (MFFGD) algorithm to address these limitations. It provides fractional-order gradient derivation and error analysis for different activation functions and loss functions within the network, simplifying the computation of traditional fractional order gradients. Additionally, by introducing a memory factor to record past gradient variations, MFFGD achieves adaptive adjustment capabilities. Comparative experiments were conducted on multiple sets of datasets with different modalities, and the results, along with theoretical analysis, demonstrate the superiority of MFFGD over other optimizers.

Optimal design of laminated thermoelectric cylindrical device based on neural network and thermo-electric-mechanical coupled model

The laminated thermoelectric cylindrical shell is favorable for solar thermal energy harvesting, but the thermo-electric-mechanical coupling field is relatively complex. An optimization design model of thermoelectric structure is required to promote its practical application. In this paper, based on variable separation method, thermo-electric-mechanical coupled model of laminated thermoelectric cylindrical device under thermal shock is constructed. Transient temperature field, thermal stress and power generation efficiency are obtained. Numerical results show that both power generation efficiency and thermal stress are affected by material parameters simultaneously. In order to analyze efficiency and structural reliability comprehensively, a fitness function is introduced in this paper. Based on the data which are collected from thermoelectric coupled model, a neural network is established and trained, which can predict and evaluate the performance of thermoelectric cylindrical shell based on the fitness function. The optimal material parameters for thermoelectric shells with high efficiency and low thermal stress are obtained. Combining neural network technique and the theoretical model developed in this paper, the computation time of numerical results can be reduced from 50 to 3 min, and this neural network optimization method has higher computational efficiency. These research results will provide guidance for the optimization design of cylindrical thermoelectric device.

The modified Mohr-Coulomb model considering softening effect and intermediate principal stress

Based on the Mohr-Coulomb model, a modified Mohr-Coulomb model that can consider both softening effect and intermediate principal stress effect was formed by introducing plastic internal variables that characterize material damage and a modified stress Rhodes angle. The modified model is numerically integrated by implicit algorithm, the physical meaning of the model parameters of the modified model is clarified, and the multivariate analysis and sensitivity analysis of the model parameters are carried out. Combined with the results of the gypsum triaxial test, the model parameters of the modified model were determined by the equation solving and RBF neural network optimization method, and the validity of the model was verified. The research results showed that: The modified model established can consider both the material softening effect and the influence of the intermediate principal stress. Compared with the traditional Mohr-Coulomb model, the calculation results of the modified model are more consistent with the test results qualitatively and quantitatively. The comprehensive use of direct determination, equation solving and RBF neural network optimization methods can effectively determine the model parameters of the modified model, and the determination process is relatively simple. It is suggested that this combined method can be used for subsequent determination of more complex constitutive model parameters. Compared with the traditional Mohr-Coulomb model, the modified model only needs one more true-triaxial test to determine the model parameters. In addition, the experience of using the parameters of the traditional Mohr-Coulomb model in a large number of engineering practices can be used to improve the model, so the modified model has great potential for engineering applications.

Optimizing Variational Quantum Neural Networks Based on Collective Intelligence

Quantum machine learning stands out as one of the most promising applications of quantum computing, widely believed to possess potential quantum advantages. In the era of noisy intermediate-scale quantum, the scale and quality of quantum computers are limited, and quantum algorithms based on fault-tolerant quantum computing paradigms cannot be experimentally verified in the short term. The variational quantum algorithm design paradigm can better adapt to the practical characteristics of noisy quantum hardware and is currently one of the most promising solutions. However, variational quantum algorithms, due to their highly entangled nature, encounter the phenomenon known as the “barren plateau” during the optimization and training processes, making effective optimization challenging. This paper addresses this challenging issue by researching a variational quantum neural network optimization method based on collective intelligence algorithms. The aim is to overcome optimization difficulties encountered by traditional methods such as gradient descent. We study two typical applications of using quantum neural networks: random 2D Hamiltonian ground state solving and quantum phase recognition. We find that the collective intelligence algorithm shows a better optimization compared to gradient descent. The solution accuracy of ground energy and phase classification is enhanced, and the optimization iterations are also reduced. We highlight that the collective intelligence algorithm has great potential in tackling the optimization of variational quantum algorithms.

A Recurrent Neural Network Optimization Method for Anticipation of Hierarchical Human Activity

A Recurrent Neural Network Optimization Method for Anticipation of Hierarchical Human Activity

Environmental Risk Assessment of the Harbin Section of the Songhua River Basin Based on Multi-Source Data Fusion

Surface water is a vital resource for human survival. However, economic and social development has resulted in significant pollutants from human activities, causing environmental pollution in watersheds. This pollution has had a profound impact on the surface water environment. However, limited studies have been conducted on the environmental risk evaluation of the watershed. In this study, we accounted for agricultural, industrial, and domestic source discharges in the districts and counties of the Harbin section of the Songhua River Basin for 2021. Data were collected from Statistical Yearbooks and governmental departments, and the characteristics of pollutant discharges in Harbin’s districts and counties were analyzed. Subsequently, we employed the Back Propagation neural network optimization method, combining remote sensing data, accounting data, pollution discharge data from each district and county, and economic and social data from the Statistical Yearbook and literature. This fusion of multiple data sources facilitated the construction of a watershed environmental risk evaluation system. The analysis considered four levels: economic and social, resource load, environmental infrastructure, and pollution discharge. Via this comprehensive evaluation, we identified the reasons for environmental risks in the water environment of the Harbin section of the Songhua River Basin. The evaluation results indicate that Nangang District, Xiangfang District, and Pingfang District face a higher risk to the water environment. Consequently, recommendations for mitigating water environment risks in these areas and across Harbin City are presented. The research methods and findings in this paper contribute valuable insights for developing control strategies to manage water quality in critically polluted areas of the Harbin section of the Songhua River Basin, providing a scientific foundation for regional river water quality management studies.

Fischer machine learning for mobile cloud computing in eHealth systems using blockchain mechanism

The Electronic Healthcare (eHealth) systems are competent to ensure effective care engineering and intensified healthcare quality which are user-friendly cache and administration, in Electronic Health Records (EHRs). For secure EHRs of Mobile Cloud-based eHealth systems, ensuring high security and data privacy, Interplanetary File System in healthcare has traditionally been concentrated. However, there has been a recent push towards achieving high quality of e-health services because blockchain-based health care applications require QoS guarantees in terms of requirements such as network latency and end-to-end delay. In this work, an Extended Validation Certification-based Fischer Neural Network Optimization (EVC-FNNO) method for secured Mobile Cloud-based E-Health Systems is proposed. With the identity being the digital certificate, the EVC is provided with the identity to the mobile cloud user who will transact in the network. In this way, the mobile cloud user is being ensured to access the ledger for the transaction. Therefore, both data privacy and security is said to be provided. Next, with Fischer Neural Network Optimization (FNNO), every authenticated mobile cloud user via EVC then possess a copy of shared ledger, therefore resolving data acquisition in cloud server and hence solving network latency. The proposed method is verified by some demonstrative examples in addressing QoS. The empirical results show that the EVC-FNNO method provides an efficient solution by validating the mobile cloud user sensitive health information with digital certificate. Security analysis proves that the EVC-FNNO method is secure. We also conduct comprehensive performance evaluations that demonstrate the high efficiency of the EVC-FNNO method in terms of end-to-end delay, network latency and data privacy, compared to the existing data sharing methods.

Automated method based on a neural network model for searching energy-efficient complex movement trajectories of industrial robot in a differentiated technological process

Purpose of the work: Researching possibility of creating a method for improving the energy efficiency of differentiated robotic technological process (RTP) in the food industry. The high rates of development of production processes robotization, including in the food industry, leading to increase the cost of electrical energy, determine the research tasks relevance in finding energy-efficient methods for controlling industrial robots.Methodology: The proposed approach is based on principles of object-oriented design and possibility of classifying robotic technological process: Applying specific model sets and methods to improve energy efficiency to individual classes. The existing possibility of energy consumption synthesizing models of robots inside differentiated robotic technological process, such as stacking loads, in reduced form, made it possible to use neural network methods to identify non-linear dependencies. At the same time, training sample for intelligent modules was formed on the basis of classical experiment planning algorithms. The synthesis of methods, models and procedures was implemented on the basis of high-level programming languages C++, MATLAB.Results: A mathematical model and automated algorithms for its synthesis are proposed, which make it possible to adjust robotic technological process simulation model taking into account its specifics and implement a method for finding the optimal parameters of its functioning. To confirm the effectiveness of proposed solution, the obtained neural network model and optimization method were tested on real robotic technological process, and the calculation of economic efficiency of proposed solution was also given.Conclusion/recommendations: The application of this approach will significantly reduce energy costs for robotic operations in the food industry.

DCOM-GNN: A Deep Clustering Optimization Method for Graph Neural Networks

Deep clustering plays an important role in data analysis, and with the prevalence of graph data nowadays, various deep clustering models on graph are constantly proposed. However, due to the lack of more adequate clustering guidance, the discriminability of feature representation learned from these models for the clustering task is limited. Therefore, for the purpose of enabling the output of these models to be more cluster-oriented, we propose a Deep Clustering Optimization Method for Graph Neural Networks (DCOM-GNN), which can be attached to the original model architecture conveniently. For DCOM-GNN, it contains two components, one is the inter-cluster distance optimization module, whose role is to further adjust the distance between clusters of the original model output rationally. Another one is the intra-cluster distance optimization module, which aims to improve the cohesiveness of the original model output. Comprehensive experiments show that the performance of various deep clustering models on graph can be significantly improved after adding DCOM-GNN.

IMPLEMENTATION OF TECHNOLOGY FOR IMPROVING THE QUALITY OF SEGMENTATION OF MEDICAL IMAGES BY SOFTWARE ADJUSTMENT OF CONVOLUTIONAL NEURAL NETWORK HYPERPARAMETERS

Background. The scientists have built effective convolutional neural networks in their research, but the issue of optimal setting of the hyperparameters of these neural networks remains insufficiently researched. Hyperparameters affect model selection. They have the greatest impact on the number and size of hidden layers. Effective selection of hyperparameters improves the speed and quality of the learning algorithm. It is also necessary to pay attention to the fact that the hyperparameters of the convolutional neural network are interconnected. That is why it is very difficult to manually select the effective values of hyperparameters, which will ensure the maximum efficiency of the convolutional neural network. It is necessary to automate the process of selecting hyperparameters, to implement a software mechanism for setting hyperparameters of a convolutional neural network. The author has successfully implemented the specified task.
 Objective. The purpose of the paper is to develop a technology for selecting hyperparameters of a convolutional neural network to improve the quality of segmentation of medical images..
 Methods. Selection of a convolutional neural network model that will enable effective segmentation of medical images, modification of the Keras Tuner library by developing an additional function, use of convolutional neural network optimization methods and hyperparameters, compilation of the constructed model and its settings, selection of the model with the best hyperparameters.
 Results. A comparative analysis of U-Net and FCN-32 convolutional neural networks was carried out. U-Net was selected as the tuning network due to its higher quality and accuracy of image segmentation. Modified the Keras Tuner library by developing an additional function for tuning hyperparameters. To optimize hyperparameters, the use of the Hyperband method is justified. The optimal number of epochs was selected - 20. In the process of setting hyperparameters, the best model with an accuracy index of 0.9665 was selected. The hyperparameter start_neurons is set to 80, the hyperparameter net_depth is 5, the activation function is Mish, the hyperparameter dropout is set to False, and the hyperparameter bn_after_act is set to True.
 Conclusions. The convolutional neural network U-Net, which is configured with the specified parameters, has a significant potential in solving the problems of segmentation of medical images. The prospect of further research is the use of a modified network for the diagnosis of symptoms of the coronavirus disease COVID-19, pneumonia, cancer and other complex medical diseases.

A Novel Fusion Method for State-of-Charge Estimation of Lithium-Ion Batteries Based on Improved Genetic Algorithm BP and Adaptive Extended Kalman Filter

The lithium-ion battery is the power source of an electric vehicle, so it is of great significance to estimate the state of charge (SOC) of lithium-ion batteries accurately to ensure vehicle safety. To improve the accuracy of the parameters of the equivalent circuit model for batteries, a second-order RC model for ternary Li-ion batteries is established, and the model parameters are identified online based on the forgetting factor recursive least squares (FFRLS) estimator. To improve the accuracy of SOC estimation, a novel fusion method, IGA-BP-AEKF, is proposed. Firstly, an adaptive extended Kalman filter (AEKF) is used to predict the SOC. Then, an optimization method for BP neural networks (BPNNs) based on an improved genetic algorithm (IGA) is proposed, in which pertinent parameters affecting AEKF estimation are utilized for BPNN training. Furthermore, a method with evaluation error compensation for AEKF based on such a trained BPNN is proposed to enhance SOC evaluation precision. The excellent accuracy and stability of the suggested method are confirmed by the experimental data under FUDS working conditions, which indicates that the proposed IGA-BP-EKF algorithm is superior, with the highest error of 0.0119, MAE of 0.0083, and RMSE of 0.0088.

SGDAT: An optimization method for binary neural networks

Stochastic gradient descent (SGD), one of the most popular neural network optimization algorithms, has a solid theoretical foundation as well as good generalization performance. However vanilla SGD performs catastrophically in Binary Neural Networks (BNNs). Many studies have identified this phenomenon without explaining its causes in depth. In this paper, we try to experimentally understand the possible reasons for this and significantly improve the performance of vanilla SGD in BNNs training by adjusting the training strategy to be comparable to Adam. We subsequently propose a new optimization method for training deep neural networks (DNNs) with binary weights. In the proposed SGD with Adaptive Threshold, referred to as SGDAT, we suppress the frequency of weights flipping by thresholds and adjust the threshold of each parameter according to the number of flipping to further reduce the network noise, stabilize the network training, and improve the network generalization ability. Also, we present a complete ablation study of the hyperparameters space, as well as experimentally analyze the impact of using adaptive thresholds. Furthermore, we conduct image classification experiments over the CIFAR10, CIFAR100 and TinyImageNet datasets using BinaryNet and ResNet-18 network structure. The experiments show that SGDAT outperforms other binary optimizers. Code is available at: https://github.com/gushan/SGDAT.

Application of Fuzzy-RBF-CNN Ensemble Model for Short-Term Load Forecasting

Accurate load forecasting (LF) plays an important role in the operation and decision-making process of the power grid. Although the stochastic and nonlinear behavior of loads is highly dependent on consumer energy requirements, that demands a high level of accuracy in LF. In spite of several research studies being performed in this field, accurate load forecasting remains an important consideration. In this article, the design of a hybrid short-term load forecasting model (STLF) is proposed. This work combines the features of an artificial neural network (ANN), ensemble forecasting, and a deep learning network. RBFNNs and CNNs are trained in two phases using the functional link artificial neural network (FLANN) optimization method with a deep learning structure. The predictions made from RBFNNs have been computed and produced as the forecast of each activated cluster. This framework is known as fuzzy-RBFNN. This proposed framework is outlined to anticipate one-week ahead load demand on an hourly basis, and its accuracy is determined using two case studies, i.e., Hellenic and Cretan power systems. Its results are validated while comparing with four benchmark models like multiple linear regression (MLR), support vector machine (SVM), ML-SVM, and fuzzy-RBFNN in terms of accuracy. To demonstrate the performance of RBF-CNN, SVMs replace the RBF-CNN regressor, and this model is identified as an ML-SVM having 3 layers.

Latency-constrained DNN architecture learning for edge systems using zerorized batch normalization

Deep learning applications have been widely adopted on edge devices, to mitigate the privacy and latency issues of accessing cloud servers. Deciding the number of neurons during the design of a deep neural network to maximize performance is not intuitive. Particularly, many application scenarios are real-time and have a strict latency constraint, while conventional neural network optimization methods do not directly change the temporal cost of model inference for latency-critical edge systems. In this work, we propose a latency-oriented neural network learning method to optimize models for high accuracy while fulfilling the latency constraint. For efficiency, we also introduce a universal hardware-customized latency predictor to optimize this procedure to learn a model that satisfies the latency constraint by only a one-shot training process. The experiment results reveal that, compared to state-of-the-art methods, our approach can well-fit the ‘hard’ latency constraint and achieve high accuracy. Under the same training settings as the original model and satisfying a 34 ms latency constraint on the ImageNet-100 dataset, we reduce GoogLeNet’s latency from 40.32 ms to 34 ms with a 0.14% accuracy reduction on the NVIDIA Jetson Nano. When coupled with quantization, our method can be further improved to only 0.04% drop for GoogLeNet. On the NVIDIA Jetson TX2, we compress VGG-19 from 119.98 ms to 34 ms and even improve its accuracy by 0.5%, and we scale GoogLeNet up from 20.27 ms to 34 ms and achieve higher accuracy by 0.78%. We also open source this framework at https://github.com/ntuliuteam/ZeroBN.

Thermodynamic, exergoeconomic, and economic analyses with multi-objective optimization of a novel liquid air energy storage coupled with an off-shore wind farm

The liquid air energy storage is a cutting-edge technology that covers the geographical drawbacks of other utility-scale energy storage alternatives. The coupling the liquid air storage system with renewable energies and heat recovery units can compensate the grid unstability and lack of their round trip efficiency, respectively. In the present paper, an off-shore wind farm including different wind turbines is assessed from technical, availability, and economic viewpoints to be combined with a liquid air energy storage unit, expolits a Kalina cycle for waste heat recovery. First, a comprehensive energy, exergy, and economic analyses are conducted for the integrated system; then, a coupling of artificial neural network and multi-objective genetic algorithm optimization method are carried out to find the optimum design condition. The findings show that around 93 MWh of the generated power in the wind site during off-peak period can be utilized to charge the system. Then, in addition to power generation in the wind farm, the reference system generates 76.4 MWh of extra power and 585 tons of domestic hot water during peak hours for peak shaving. Based on the optimization results, the round trip efficiency and total cost rate are 44.7% and 476 $/hr, repectively. The result of economic analysis demonstrated that the Enercon E-126 EP4 is the most cost-effective wind turbine for the proposed system, resulting in a period of 5.5 years. It has been also resulted that the existence of the LAES-KC system can half the paybacks.

Optimization for a New XY Positioning Mechanism by Artificial Neural Network-Based Metaheuristic Algorithms.

This paper devotes a new method in modeling and optimizing to handle the optimization of the XY positioning mechanism. The fitness functions and constraints of the mechanism are formulated via proposing a combination of artificial neural network (ANN) and particle swarm optimization (PSO) methods. Next, the PSO is hybridized with the grey wolf optimization, namely PSO-GWO, which is applied to three scenarios in handling the single objective function. In order to search the multiple functions for the mechanism, the multiobjective optimization genetic algorithm (MOGA) is applied to the last scenario. The achieved results showed that the fitness functions are well-formulated using the PSO-based ANN method. In the scenario 1, the stroke achieved by the PSO-GWO (1852.9842 μm) is better than that gained from the GWO (1802.8087 μm). In the scenarios 2, the stress gained from the PSO-GWO (243.3183 MPa) is lower than that achieved from the GWO (245.0401 MPa). In the scenario 3, the safety factor retrieved from the PSO-GWO (1.9767) is greater than that achieved from the GWO (1.9278). In the scenario 4, by using MOGA, the optimal results found that the stroke is about (1741.3 μm) and the safety factor is 1.8929. The prediction results are well-fitted with the numerical and experimental verifications. The results of this paper are expected to facilitate the synthesis and analysis of compliant mechanisms and related engineering designs.

Retracted: Design and Application of BP Neural Network Optimization Method Based on SIWSPSO Algorithm

Retracted: Design and Application of BP Neural Network Optimization Method Based on SIWSPSO Algorithm

Swarm intelligence for new materials

With the help of deep learning technology, constructing the neural network model between structures and properties of materials is one of important methods to speed up the new materials discovery and design. However, both the training efficiency and accuracy of model play important roles in application of machine learning on materials. At present, the commonly used traditional neural network model algorithm in materials is single, easy to fall into local optimization, and difficult to converge in the training process, and the training model has a low prediction power. In this paper, a novel neural network optimization method based on swarm algorithms is proposed to discover better parameters in developing new materials. In this neural network model, particle swarm and Bayesian optimization are used to improve the crossover, mutation and selection strategies of genetic algorithm. Using advantages of each algorithm to realize the optimal parameters selection of the neural network and network optimization, and to improve the prediction ability of the neural network model. By comparing energies predicted by traditional networks and the proposed network, more stable models with high efficiency and precision are trained.

Dataflow Optimization through Exploring Single-Layer and Inter-Layer Data Reuse in Memory-Constrained Accelerators

Off-chip memory access has become the performance and energy bottleneck in memory-constrained neural network accelerators. To provide a solution for the energy efficient processing of various neural network models, this paper proposes a dataflow optimization method for modern neural networks by exploring the opportunity of single-layer and inter-layer data reuse to minimize the amount of off-chip memory access in memory-constrained accelerators. A mathematical analysis of three inter-layer data reuse methods is first presented. Then, a comprehensive exploration to determine the optimal data reuse strategy from single-layer and inter-layer data reuse approaches is proposed. The result shows that when compared to the existing single-layer-based exploration method, SmartShuttle, the proposed approach can achieve up to 20.5% and 32.5% of off-chip memory access reduction for ResNeXt-50 and DenseNet-121, respectively.