Training Process Of Neural Network Research Articles

BP Neural Network-Enhanced System for Employment and Mental Health Support for College Students

This paper employs BP Neural Network (BPNN) theory to evaluate innovation and entrepreneurship education in universities. It utilizes students' evaluation indexes as input vectors and determines the number of hidden layer neurons. Experimental results serve as output vectors. The BPNN method proves reasonable and feasible for vocational education course evaluation, exhibiting a 14.96% higher accuracy than traditional genetic algorithms. The paper discusses the model, configuration, characteristics, training process, algorithm enhancement, and limitations of neural networks, followed by an introduction to genetic algorithms. Through analysis of principles, basic operations, and common operators, it establishes a theoretical foundation for subsequent discussions.

Theory guided Lagrange programming neural network for subsurface flow problems

A deep learning model can perform efficient uncertainty quantification (UQ) for reservoir flow with uncertain model parameters, but usually requires large amounts of training data to ensure accuracy. However, the cost of obtaining large amounts of data is prohibitive, and the performance will deteriorate if sufficient training data is lacking. Alternatively, more interpretable neural networks with embedded physical laws have recently been used to solve partial differential equations as well as to solve UQ problems. This approach has received a lot of attention due to its low data volume requirements and its adherence to the laws of physics during the training process. In this paper, we propose a theory-guided framework based on a bilevel programming model with hard constraints to embed physical meaning in the model. Theory guided Lagrange programming neural network (TGLPNN) combines the method of Lagrange programming neural network approach where physical laws such as stochastic partial differential equations and boundary conditions are incorporated into the training process of a convolutional neural network. At the same time, the upper-level variables are iteratively optimized. The method based on Lagrange programming neural network inherently embeds physical laws in the network. Practical applications have shown that TGLPNN can provide higher prediction accuracy compared to state-of-the-art physics-driven methods and improved efficiency compared to numerical methods.

PerfTop: Towards performance prediction of distributed learning over general topology

Distributed learning with multiple GPUs has been widely adopted to accelerate the training process of large-scale deep neural networks. However, misconfiguration of the GPU clusters with various communication primitives and topologies could potentially diminish the gains in parallel computation and lead to significant degradation in training efficiency. Predicting the performance of distributed learning enables service providers to identify potential bottlenecks beforehand. In this work, we propose a Performance prediction framework over General Topologies, called PerfTop, for accurate estimation of per-iteration execution time. The main strategy is to integrate computation time prediction with an analytical model to map the nonlinearity in communication and fine-grained computation-communication patterns. This enables accurate prediction of a variety of neural network models over general topologies, such as tree, hierarchical, and exponential. Our extensive experiments show that PerfTop outperforms existing methods in estimating both computation and communication time, particularly for communication, surpassing the existing methods by over 45%. Meanwhile, it achieves an accuracy of above 85% in predicting the execution time over general topologies compared to simple topologies such as star and ring from the previous works.

Classification of plant leaf diseases using deep neural networks in color and grayscale images

For a long time, plants have shown a crucial role in our life, manufacturing and industry. However, a large number of diseases have significantly affected to the production of plants. The detection and diagnosis of the diseases are necessary to improve the production of plants. In recent years, the automatic classification of plant diseases using artificial intelligence and computer vision have attracted a huge number of researches. The paper presents a classification method of plant leaf diseases using various Deep Neural Networks (e.g., Alexnet, Resnet-50, Densenet-121). The color features have significantly affected the classification accuracy. Therefore, we analyzed and compared the classification accuracy on two public datasets (Plant village leaf datasets) that consist of color and grayscale images. The method obtained the classification accuracy of 98.08% and 92% on color and grayscale images, respectively. The obtained results showed the effectiveness of the method. Based on the obtained results, the impact of color features to the classification accuracy of various Deep Neural Networks is analyzed in the paper. Moreover, the paper compares the performance of various optimization algorithms during the training process of deep neural networks to classify leaf diseases.

Integration of Artificial Intelligence for Enhanced Coordination of DOCR Protection in Distributed Generation Systems

Distributed generation (DG) is an approach that involves adding decentralized power generation within a distribution network. Distributed generation systems can reduce transmission losses, increase the reliability of energy supply, minimize carbon emissions, and enable the active participation of consumers in energy production. However, with the increase in distributed generation, electric power systems face new challenges in maintaining operational reliability and safety. Disruptions such as short circuits or overcurrent can occur in the system, and appropriate protective responses are required to protect the power grid from more significant damage. The addition of DG also causes the short circuit current to vary and results in system protection coordination having to be redone. Carrying out coordination will take a long time. This research uses modeling and simulation of a distributed generation system with various operating conditions and works adaptively according to changes in the system due to the addition of DG. The results obtained from the simulation are used in neural network training to study the relationship patterns between directional overcurrent relays (DOCR) parameters and system operating conditions. The backpropagation algorithm is used in the Artificial Neural Network (ANN) training process. The training process utilizes the maximum Short Circuit Current (ISC) input obtained through generation, fault location, and fault type. Time Dial Setting (TDS) and Ipickup values are used as ANN training targets. After testing, the results obtained are in accordance with the target data. The efficacy of this method is further demonstrated through ETAP simulations, which confirm that ANN is a suitable approach for modeling adaptive and optimal relay coordination systems.

End-to-End Differentiable Physics Temperature Estimation for Permanent Magnet Synchronous Motor

Differentiable physics is an approach that effectively combines physical models with deep learning, providing valuable information about physical systems during the training process of neural networks. This integration enhances the generalization ability and ensures better consistency with physical principles. In this work, we propose a framework for estimating the temperature of a permanent magnet synchronous motor by combining neural networks with the differentiable physical thermal model, as well as utilizing the simulation results. In detail, we first implement a differentiable thermal model based on a lumped parameter thermal network within an automatic differentiation framework. Subsequently, we add a neural network to predict thermal resistances, capacitances, and losses in real time and utilize the thermal parameters’ optimized empirical values as the initial output values of the network to improve the accuracy and robustness of the final temperature estimation. We validate the conceivable advantages of the proposed method through extensive experiments based on both synthetic data and real-world data and then provide some further potential applications.

A New Approach to Structural Damage Identification Based on Power Spectral Density and Convolutional Neural Network

In the field of structural health monitoring, vibration-based damage identification remains a formidable challenge. Key to this challenge is the establishment of a reliable association between observed vibration characteristics and the actual state of structural damage (e.g. stiffness reduction). This association not only accurately indicates the presence of damage, but also the location and severity of the damage. To solve this complex pattern identification problem, a large number of approaches, including deep learning, have emerged in recent years. In this paper, we propose a new structural damage identification method that utilizes the vibration information of the structure and a convolutional neural network based on Alex NET improvement. The method consists of calculating the acceleration response power spectral density of damaged and undamaged structures under impact loading separately, and then making a difference between the two power spectral data, and subsequently introducing these power spectral difference data into the convolutional neural network for training. The use of power spectral density analysis as a preprocessing step converts the time-domain signals into frequency-domain signals, and this conversion allows the convolutional neural network to capture and learn from the specific frequency characteristics of the data, thus facilitating the learning process of the neural network model. In this paper, the effectiveness of the method is critically evaluated through numerical simulation and experimental validation, and 3% and 5% noise are added to the numerical study to test the robustness of the method. During the convolution neural network training process, the optimal training mean squared error (MSE) is [Formula: see text] in the case of no noise addition; the optimal training MSE is [Formula: see text] in the case of noise addition. Both the results of simulations and experiments confirm the high accuracy and good robustness of the method in localizing structural damage.

Neural Network-based Pattern Recognition in the Framework of Edge Computing

Neural network (NN) model has been widely used in pattern recognition (PR), speech recognition, image processing and other fields, but its application in edge computing (EC) environment faces performance and energy consumption problems. This article first introduced the basic structure and training process of NN, including backpropagation algorithms. Then, this article presented a NN modeling approach based on EC, including NN model compression, distributed NN model and knowledge distillation approach. Finally, this article implemented a PR model for the MNIST (Mixed National Institute of Standards and Technology database) dataset and analyzed the experimental results. The experimental outcomes indicated that the presented approach can significantly enhance the performance of the NN model in the EC environment, while ensuring a high recognition accuracy. The NN modeling approach based on EC can reduce the amount of computation and storage of the NN, thus improving the operating efficiency of the NN in the EC environment by 6%-12%. The NN modeling approach based on EC can optimize the performance and efficiency of the NN model in the EC environment, and provide new ideas and approaches for the application of NN in the EC environment.

Improving multilayer perceptron neural network using two enhanced moth-flame optimizers to forecast iron ore prices

Abstract The quality of the output produced by the multi-layer perceptron neural network depends on the careful selection of its weights and biases. The gradient descent technique is commonly used for choosing MLP’s optimal configuration, but it can suffer from being stuck in local optima and slow convergence toward promising regions in the search space. In this article, we propose two new optimization algorithms based on the moth-flame optimization algorithm (MFO), which mimics moths’ special navigation methods at night. We use these algorithms to enhance the performance of the training process of the MLP neural network. To demonstrate the effectiveness of our approach, we apply it to the problem of predicting iron ore prices, which plays an important role in the continuous development of the steel industry. We use a large number of features to predict the iron ore price, and we select a promising set of features using two feature reduction methods: Pearson’s correlation and a newly proposed categorized correlation. Surprisingly, new features not mentioned in the literature are discovered, and some are discarded. The time series dataset used has been extracted from several sources and pre-processed to fit the proposed model. We compare our two proposed MFO algorithms, the roulette wheel moth-flame optimization algorithm and the global best moth-flame optimization algorithm, against four swarm intelligence algorithms and five classical machine learning techniques when predicting the iron ore price. The results acquired indicate the superior performance of the suggested algorithms concerning prediction accuracy, root-mean-square error, mean-square error, average absolute relative deviation, and mean absolute error. Overall, our work presents a promising approach for improving the performance of MLP neural networks, and it demonstrates its effectiveness in the challenging problem of predicting iron ore prices.

Optimizing neural network training with Genetic Algorithms

In modern society, computer plays an important role among all human beings. Through the increasing development of technology, some problems happened gradually. In order to solve and regenerate the country, individuals should test their strengths. This paper discusses how to use genetic algorithms to optimize neural network training. As an important tool of machine learning, neural networks have made remarkable achievements in dealing with complex tasks. However, the training process of neural networks involves a lot of hyperparameter adjustment and weight optimization, which often requires a lot of time and computing resources. In order to improve the efficiency and performance of neural network training, humans should introduce genetic algorithms as an optimization method. Experiments are conducted on several common datasets to compare the performance of neural network training with Genetic Algorithm optimization against the traditional method. The results indicate that using Genetic Algorithms significantly improves the convergence speed and performance of neural networks while reducing the time and effort spent on hyperparameter tuning. Neural networks optimized using the Genetic Algorithm outperform their counterparts trained under the same time frame.

PKDGAN: Private Knowledge Distillation With Generative Adversarial Networks

The deployment of deep learning applications has to address the increasing privacy concerns when using private and sensitive data for training. A conventional deep learning model is prone to privacy attacks that can recover the sensitive information from either model parameters or accesses to the inference model. Recently, differential privacy (DP) has been proposed to offer provable privacy guarantees by randomizing the training process of neural networks. However, many approaches tend to provide the worst case privacy protection for model publishing, inevitably impairing the accuracy of the trained models. Thus, we present a novel private knowledge transfer strategy, where the private teacher trained on sensitive data is not publicly accessible but the student models can be released with privacy guarantees. In this paper, a three-player (teacher-student-discriminator) learning framework, Private Knowledge Distillation with Generative Adversarial Networks (PKDGAN), is proposed, where the student acquires the distilled knowledge from the teacher and is trained with the discriminator to generate similar outputs as the teacher. Moreover, a cooperative learning strategy is also suggested to support the collective training of multiple students against the discriminator when each student is with insufficient unlabelled training data. To enforce rigorous privacy guarantees, PKDGAN applies a Rényi differential privacy mechanism throughout the training process, and use it with a moment accountant technique to track the privacy cost. PKDGAN allows students to be trained with unlabelled public data and very few epochs, which avoids the exposure of training data while ensuring model performance. In the experiments, PKDGAN is found to have consistently good performance on various datasets (MNIST, SVHN, CIFAR-10, and Market-1501). When compared to prior works [1], [2], PKDGAN exhibits 5-82% accuracy loss improvement without compromising any privacy guarantee.

Biobjective gradient descent for feature selection on high dimension, low sample size data.

Even though deep learning shows impressive results in several applications, its use on problems with High Dimensions and Low Sample Size, such as diagnosing rare diseases, leads to overfitting. One solution often proposed is feature selection. In deep learning, along with feature selection, network sparsification is also used to improve the results when dealing with high dimensions low sample size data. However, most of the time, they are tackled as separate problems. This paper proposes a new approach that integrates feature selection, based on sparsification, into the training process of a deep neural network. This approach uses a constrained biobjective gradient descent method. It provides a set of Pareto optimal neural networks that make a trade-off between network sparsity and model accuracy. Results on both artificial and real datasets show that using a constrained biobjective gradient descent increases the network sparsity without degrading the classification performances. With the proposed approach, on an artificial dataset, the feature selection score reached 0.97 with a sparsity score of 0.92 with an accuracy of 0.9. For the same accuracy, none of the other methods reached a feature score above 0.20 and sparsity score of 0.35. Finally, statistical tests validate the results obtained on all datasets.

Research on lightweight GPR road surface disease image recognition and data expansion algorithm based on YOLO and GAN

The aim of this paper is to improve the accuracy and efficiency of ground penetrating Radar (GPR) detection of internal road surface disease images. Based on the YOLOv4 target detection algorithm, this study introduces MobileNetV2 and CBAM attention mechanism, and combines the Focal loss confidence loss function to iterate the model, so as to design an efficient and lightweight GPR pavement disease image recognition algorithm, MC-YOLOv4. At the same time, in order to alleviate the problem of data scarcity in GPR, we redesign an unsupervised generative adversarial neural network based on self-attention mechanism, namely SAGAN-W. Experiments show that MC-YOLOv4 not only has faster reasoning ability, but also occupies only 23% of the memory of YOLOv5-S. After using the SAGAN data augmentation, the mAP@ .0.5 evaluation index is further improved by 2.63%, and the collapse and mode collapse that may occur in the training process of the traditional Convolutional Generative Adversarial Neural Network (DCGAN) are avoided. After introducing the Focal loss confidence loss function to participate in the training, It significantly improves the imbalance between the precision and recall of the detection model, and this scheme is verified and supported by real scenes. The experimental results show that the proposed method has significant advantages in automatic detection and data expansion of lightweight GPR pavement invisible diseases, which has a wide range of application value and research significance.

Pseudo-partial-derivative information-driven adaptive fault-tolerant tracking control for discrete-time systems

The fault-tolerant tracking control problem is studied for the discrete-time systems with actuator faults. To lessen adverse impacts of actuator fault, a PPD information-driven fault estimation algorithm is established to adaptively estimate actuator fault information online, which avoids the additional construction and training process of neural network. With the aid of the adaptive fault compensation, a model-free adaptive fault-tolerant tracking control algorithm is constructed to ensure that the expected output reference trajectory can be tracked by system output. Moreover, only input and output data are employed throughout the design process, system dynamics are not demanded. Ultimately, the availability of developed strategy is proved through a simulation.

Magnetic flux leakage defect size estimation method based on physics-informed neural network.

Magnetic flux leakage (MFL) is a magnetic method of non-destructive testing for in-pipe defect detection and sizing. Despite the fact that recent developments in machine learning have revolutionized disciplines like MFL defect size estimation, the most current methods for quantifying pipeline defects are primarily data-driven, which may violate the underlying physical knowledge. This paper proposes a physics-informed neural network-based method for MFL defect size estimation. The training process of neural network is guided by the MFL data and the physical constraints that is mathematically represented by the magnetic dipole model. We use synthetic MFL data produced by a virtual MFL testing of pipeline defects to validate the proposed method through a comparison to purely data-driven neural networks and support vector machines. The findings imply that the physics-informed strategy can both improve predictive accuracy and mitigate physical violations in MFL testing, providing us with a better knowledge of how neural networks perform in defect size estimation. This article is part of the theme issue 'Physics-informed machine learning and its structural integrity applications (Part 2)'.

On the use of artificial intelligence in predicting the compressive strength of various cardboard packaging

AbstractArtificial intelligence is increasingly used in various branches of engineering. In this article, artificial neural networks are used to predict the crush resistance of corrugated packaging. Among the analysed packages were boxes with ventilation openings, packages with perforations and typical flap boxes, which make the proposed estimation method very universal. Typical shallow feedforward networks were used, which are perfect for regression problems, mainly when the set of input and output parameters is small, so no complicated architecture or advanced learning techniques are required. The input parameters of the neural networks are selected so as to take into account not only the material used for the production of the packaging but also the dimensions of the box and the impact of ventilation holes and perforations on the load capacity of individual walls of the packaging. In order to maximize the effectiveness of neural network training process, the group of input parameters was changed so as to eliminate those to which the sensitivity of the model was the lowest. This allowed the selection of the optimal configuration of training pairs for which the estimation error was on the acceptable level. Finally, models of neural networks were selected, for which the training and testing error did not exceed 10%. The demonstrated effectiveness allows us to conclude that the proposed set of universal input parameters is suitable for efficient training of a single neural network model capable of predicting the compressive strength of various types of corrugated packaging.

Physics-Guided V-Net: An Efficient Surrogate Model for Well Placement Optimization in Petroleum Reservoirs

Summary Determination of the optimal well placement strategy in oil or gas fields is crucial for economic reservoir development. The optimization process, however, can be computationally intensive as a result of the potentially high-dimensional search space and the expensive numerical simulation. In this study, machine-learning-based surrogate models are constructed as efficient alternatives to numerical simulators to accelerate the optimization process. A V-Net neural network architecture is used with features of skip connections, 3D convolutional filters, and a residual learning structure to handle 3D parameter fields effectively. Physical guidance is incorporated into the neural network training process by adding governing equation constraints to the loss function in the discretized form, resulting in a physics-guided machine learning architecture: PgV-Net. Well placement optimization problems in a 3D oil-bearing formation with strong porosity and permeability heterogeneity are studied using the PgV-Net-based surrogate model along with genetic algorithms (GAs). Three optimization problems with increased complexity are solved, and the results are compared with regular approaches using a numerical simulator. Good agreement between the two approaches is observed, and the computational efficiency improves dramatically (up to 30 times). The proposed PgV-Net neural network training architecture reduces the requirement of expensive training data and can be used for more challenging problems such as multiphase flow modeling.

Manipulating the loss calculation to enhance the training process of physics-informed neural networks to solve the 1D wave equation

Manipulating the loss calculation to enhance the training process of physics-informed neural networks to solve the 1D wave equation

Searching for Promisingly Trained Artificial Neural Networks

Assessing the training process of artificial neural networks (ANNs) is vital for enhancing their performance and broadening their applicability. This paper employs the Monte Carlo simulation (MCS) technique, integrated with a stopping criterion, to construct the probability distribution of the learning error of an ANN designed for short-term forecasting. The training and validation processes were conducted multiple times, each time considering a unique random starting point, and the subsequent forecasting error was calculated one step ahead. From this, we ascertained the probability of having obtained all the local optima. Our extensive computational analysis involved training a shallow feedforward neural network (FFNN) using wind power and load demand data from the transmission systems of the Netherlands and Germany. Furthermore, the analysis was expanded to include wind speed prediction using a long short-term memory (LSTM) network at a site in Spain. The improvement gained from the FFNN, which has a high probability of being the global optimum, ranges from 0.7% to 8.6%, depending on the forecasting variable. This solution outperforms the persistent model by between 5.5% and 20.3%. For wind speed predictions using an LSTM, the improvement over an average-trained network stands at 9.5%, and is 6% superior to the persistent approach. These outcomes suggest that the advantages of exhaustive search vary based on the problem being analyzed and the type of network in use. The MCS method we implemented, which estimates the probability of identifying all local optima, can act as a foundational step for other techniques like Bayesian model selection, which assumes that the global optimum is encompassed within the available hypotheses.

Training of Feed-Forward Neural Networks by Using Optimization Algorithms Based on Swarm-Intelligent for Maximum Power Point Tracking.

One of the most used artificial intelligence techniques for maximum power point tracking is artificial neural networks. In order to achieve successful results in maximum power point tracking, the training process of artificial neural networks is important. Metaheuristic algorithms are used extensively in the literature for neural network training. An important group of metaheuristic algorithms is swarm-intelligent-based optimization algorithms. In this study, feed-forward neural network training is carried out for maximum power point tracking by using 13 swarm-intelligent-based optimization algorithms. These algorithms are artificial bee colony, butterfly optimization, cuckoo search, chicken swarm optimization, dragonfly algorithm, firefly algorithm, grasshopper optimization algorithm, krill herd algorithm, particle swarm optimization, salp swarm algorithm, selfish herd optimizer, tunicate swarm algorithm, and tuna swarm optimization. Mean squared error is used as the error metric, and the performances of the algorithms in different network structures are evaluated. Considering the results, a success ranking score is obtained for each algorithm. The three most successful algorithms in both training and testing processes are the firefly algorithm, selfish herd optimizer, and grasshopper optimization algorithm, respectively. The training error values obtained with these algorithms are 4.5 × 10-4, 1.6 × 10-3, and 2.3 × 10-3, respectively. The test error values are 4.6 × 10-4, 1.6 × 10-3, and 2.4 × 10-3, respectively. With these algorithms, effective results have been achieved in a low number of evaluations. In addition to these three algorithms, other algorithms have also achieved mostly acceptable results. This shows that the related algorithms are generally successful ANFIS training algorithms for maximum power point tracking.