Gradient Backpropagation Research Articles

A generic physics-informed neural network-based framework for reliability assessment of multi-state systems

In this paper, we develop a generic physics-informed neural network (PINN)-based framework to assess the reliability of multi-state systems (MSSs). The proposed framework follows a two-step procedure. In the first step, we recast the reliability assessment of MSS as a machine learning problem using the framework of PINN. A feedforward neural network with two individual loss groups is constructed to encode the initial condition and the state transitions governed by ordinary differential equations in MSS, respectively. Next, we tackle the problem of high imbalance in the magnitudes of back-propagated gradients from a multi-task learning perspective and establish a continuous latent function for system reliability assessment. Particularly, we regard each element of the loss function as an individual learning task and project a task’s gradient onto the norm plane of any other task with a conflicting gradient by taking the projecting conflicting gradients (PCGrad) method. We demonstrate the applications of the proposed framework for MSS reliability assessment in a variety of scenarios, including time-independent or dependent state transitions, where system scales increase from small to medium. The computational results indicate that PINN-based framework reveals a promising performance in MSS reliability assessment and incorporation of PCGrad into PINN substantially improves the solution quality and convergence speed of the algorithm.

Integrated optimization model for hierarchical service network design and passenger assignment in an urban rail transit network: A Lagrangian duality reformulation and an iterative layered optimization framework based on forward-passing and backpropagation

This study solves an integrated operational problem regarding hierarchical service network design and passenger assignment for urban rail transit systems. We propose an innovative nonlinear programming model for determining the number of stocking trains at each depot, number of operating trains on each line, and line-based service frequency and capacity. Given a certain passenger demand matrix, this model simultaneously determines the system-optimal path flow while assigning passengers to lines to minimize the passenger total travel cost. The proposed nonlinear programming model is then reformulated based on Lagrangian duality as two resource allocation sub-problems represented as artificial neural networks. The forward pass of the train flow sequentially assigns train resources to candidate depots and lines, and the forward pass of the passenger flow sequentially assigns the passenger demand to candidate paths and links. The solution can be improved by backpropagation of the first-order gradients and re-assignment of the train resources and passenger demand with updated weights between different layers under the proposed layered optimization framework. A comparative analysis indicates that the proposed solution approach can obtain an approximate optimal solution for the integrated optimization model, thereby providing an optimized operational hierarchical service plan and system-optimal passenger assignment results. The proposed methodology and solution approach are evaluated on a simple network case and Beijing Metro Network case.

Narx Neural Networks Models for Prediction of Standardized Precipitation Index in Central Mexico

Some of the effects of climate change may be related to a change in patterns of rainfall intensity or scarcity. Therefore, humanity is facing environmental challenges due to an increase in the occurrence and intensity of droughts. The forecast of droughts can be of great help when trying to reduce the adverse effects that the scarcity of water brings, particularly in agriculture. When evaluating the conditions of water scarcity, as well as in the identification and characterization of droughts, the use of predictive models of drought indices could be a very useful tool. In this research, the utility of Artificial Neural Networks with exogenous inputs was tested, with the aim of predicting the monthly Standardized Precipitation Index in 4 regions (Semi-desert, Highlands, Canyons and Mountains) of north-central México using predictor data from 1979 to 2014. The best model was found using the scaled conjugate gradient backpropagation algorithm as the optimization method and was set to the following architecture: 6-25-1 network. The correlation coefficient of predicted and observed Standardized Precipitation Index values for the test dataset was between 0.84 and 0.95. As a result, the Artificial Neural Network models performed successfully in predicting Standardized Precipitation Index at the four analyzed regions. The developed and tested Artificial Neural Network models in this research suggest remarkable prediction abilities of the monthly Standardized Precipitation Index in the study region.

A Study on the Prediction of Compressive Strength of Self-Compacting Recycled Aggregate Concrete Utilizing Novel Computational Approaches.

A considerable amount of discarded building materials are produced each year worldwide, resulting in ecosystem degradation. Self-compacting concrete (SCC) has 60–70% coarse and fine particles in its composition, so replacing this material with another waste material, such as recycled aggregate (RA), reduces the cost of SCC. This study compares novel Artificial Neural Network algorithm techniques—Levenberg–Marquardt (LM), Bayesian regularization (BR), and Scaled Conjugate Gradient Backpropagation (SCGB)—to estimate the 28-day compressive strength (f’c) of SCC with RA. A total of 515 samples were collected from various published papers, randomly splitting into training, validation, and testing with percentages of 70, 10 and 20. Two statistical indicators, correlation coefficient (R) and mean squared error (MSE), were used to assess the models; the greater the R and lower the MSE, the more accurate the algorithm. The findings demonstrate the higher accuracy of the three models. The best result is achieved by BR (R = 0.91 and MSE = 43.755), while the accuracy of LM is nearly the same (R = 0.90 and MSE = 48.14). LM processes the network in a much shorter time than BR. As a result, LM and BR are the best models in forecasting the 28 days f’c of SCC having RA. The sensitivity analysis showed that cement (28.39%) and water (23.47%) are the most critical variables for predicting the 28-day compressive strength of SCC with RA, while coarse aggregate contributes the least (9.23%).

Adaptive stochastic resonance based convolutional neural network for image classification

In this paper, we exploit the adaptive stochastic resonance effect in the convolutional neural network with threshold activation functions for enabling the back-propagation gradient computation. During training, the injection of noise into the threshold activation function makes it to be differentiable at forward pass, thus exact gradients of the loss function with respect to network parameters including the injected noise level can smoothly propagate at backforward pass. This training method effectively uses the non-zero noise to improve the performance of the designed convolutional neural network. In the testing phase, the differentiable activation function is decoupled into a set of threshold functions driven by mutually independent noise components, which endows a hardware-friendly feature of the designed neural network. Experiments show that the designed low-precision convolutional neural network has a comparable performance with the full-precision one with ReLU activation functions on two benchmark data sets. These results also extend the potential application of adaptive stochastic resonance into the convolutional neural network for image classification.

Experimental study and implementation of supervised machine learning algorithm to predict the flowability of two-phase water-oil in pipeline

In the middle-stream petroleum industry, the concurrent flow of oil and water in pipes is a typical occurrence. Several factors have been reported to influence the flowability of two-phase water-oil in pipeline. However, the non-linear relationship between these factors and the flowability of the two-phase water-oil is not yet known. The understanding of these relationships could help in developing models that might be employed in predicting flow behavior. In this study, two-phase water-oil flowability was experimentally investigated and modeled using supervised machine learning algorithms. The performance of each of the models was optimized by evaluating the model with the best-hidden neuron. For the Levenberg-Marquardt backpropagation, Bayesian regularization backpropagation, and scaled conjugate gradient backpropagation networks, the best topologies of 4-8-4, 4-9-4, and 4-10-4 were obtained, and coefficient of determination (R2) of 0.998, 0.999, 0.996 respectively. The predicted kinematic viscosity, dynamic viscosity, pressure drop, and power consumption were consistent with the observed values. The sensitivity analysis revealed that the temperature has the most significant effect on the predicted output. The robustness of using the supervised machine learning technique in modeling the flowability of two-phase water-crude oil mixes in pipelines based on the relationship between the predictors and the targeted variables was demonstrated.

Prediction of Splitting Tensile Strength of Self-Compacting Recycled Aggregate Concrete Using Novel Deep Learning Methods

The composition of self-compacting concrete (SCC) contains 60–70% coarse and fine aggregates, which are replaced by construction waste, such as recycled aggregates (RA). However, the complexity of its structure requires a time-consuming mixed design. Currently, many researchers are studying the prediction of concrete properties using soft computing techniques, which will eventually reduce environmental degradation and other material waste. There have been very limited and contradicting studies regarding prediction using different ANN algorithms. This paper aimed to predict the 28-day splitting tensile strength of SCC with RA using the artificial neural network technique by comparing the following algorithms: Levenberg–Marquardt (LM), Bayesian regularization (BR), and Scaled Conjugate Gradient Backpropagation (SCGB). There have been very limited and contradicting studies regarding prediction by using and comparing different ANN algorithms, so a total of 381 samples were collected from various published journals. The input variables were cement, admixture, water, fine and coarse aggregates, and superplasticizer; the data were randomly divided into three sets—training (60%), validation (10%), and testing (30%)—with 10 neurons in the hidden layer. The models were evaluated by the mean squared error (MSE) and correlation coefficient (R). The results indicated that all three models have optimal accuracy; still, BR gave the best performance (R = 0.91 and MSE = 0.2087) compared with LM and SCG. BR was the best model for predicting TS at 28 days for SCC with RA. The sensitivity analysis indicated that cement (30.07%) was the variable that contributed the most to the prediction of TS at 28 days for SCC with RA, and water (2.39%) contributed the least.

Machine Learning Techniques for Automatic Detection of Sickle Cell Anemia using Adaptive Thresholding and Contour-based Segmentation Method

Automatic diagnosis of diseases in the medical field using image processing techniques has evolved tremendously in recent times. Sickle cell anemia (SCA) is a kind of disease connected with red blood cells (RBCs) present in the human body in which deformation of cells take place. The purpose of this work is to propose an automatic image processing technique for the detection of this disease from microscopic blood images. This paper mainly focuses on automatic detection of SCA using a novel segmentation method encompassing local adaptive thresholding and active contour-based algorithm. For the detection of sickle cells, supervised classifiers such as Artificial Neural Network (ANN) and Support Vector Machine (SVM) are used. Here, geometric features of healthy and unhealthy RBCs are calculated and applied to these classifiers. In this approach, performance is found slightly greater in SVM classifier than the ANN classifier trained with scaled conjugate gradient back-propagation (BP) algorithm and with hidden layer of ten neurons. The proposed approach achieves a maximum of 99.2% accuracy with SVM classifier. The performance is also studied for seven different training algorithms in the ANN classifier by varying the numbers of hidden layer neurons. Comparative analysis of the performances of these algorithms shows that, resilient BP algorithm and 10 numbers of hidden neurons gave moderately better performance in ANN with 99% accuracy. ANN and SVM classifier with adaptive thresholding and active contour technique is an efficient approach for the classification of patients with SCA.

Inverse Design of Photonic Devices with Strict Foundry Fabrication Constraints

We introduce a new method for inverse design of nanophotonic devices which guarantees that resulting designs satisfy strict length scale constraints - including minimum width and spacing constraints required by commercial semiconductor foundries. The method adopts several concepts from machine learning to transform the problem of topology optimization with strict length scale constraints to an unconstrained stochastic gradient optimization problem. Specifically, we introduce a conditional generator for feasible designs and adopt a straight-through estimator for backpropagation of gradients to a latent design. We demonstrate the performance and reliability of our method by designing several common integrated photonic components.

Detecting land use changes using hybrid machine learning methods in the Australian tropical regions

The present study evaluates the application of the hybrid machine learning methods to detect changes of land use with a focus on agricultural lands through remote sensing data processing. Two spectral images by Landsat 8 were applied to train and test the machine learning model. Feed forward neural network classifier was utilized as the machine learning model in which two evolutionary algorithms including particle swarm optimization and invasive weed optimization were applied for the training process. Moreover, three conventional training methods including Levenberg–Marquardt back propagation (LM), Scaled conjugate gradient backpropagation (SCG) and BFGS quasi-Newton backpropagation (BFG) were used for comparing the robustness and reliability of the evolutionary algorithms. Based on the results in the case study, evolutionary algorithms are not a reliable method for detecting changes through the remote sensing analysis in terms of accuracy and computational complexities. Either BFG or LM is the best method to detect the agricultural lands in the present study. BFG is slightly more robust than the LM method. However, LM might be preferred for applying in the projects due to low computational complexities.

Successfully and efficiently training deep multi-layer perceptrons with logistic activation function simply requires initializing the weights with an appropriate negative mean

The vanishing gradient problem (i.e., gradients prematurely becoming extremely small during training, thereby effectively preventing a network from learning) is a long-standing obstacle to the training of deep neural networks using sigmoid activation functions when using the standard back-propagation algorithm. In this paper, we found that an important contributor to the problem is weight initialization.We started by developing a simple theoretical model showing how the expected value of gradients is affected by the mean of the initial weights. We then developed a second theoretical model that allowed us to identify a sufficient condition for the vanishing gradient problem to occur. Using these theories we found that initial back-propagation gradients do not vanish if the mean of the initial weights is negative and inversely proportional to the number of neurons in a layer.Numerous experiments with networks with 10 and 15 hidden layers corroborated the theoretical predictions: If we initialized weights as indicated by the theory, the standard back-propagation algorithm was both highly successful and efficient at training deep neural networks using sigmoid activation functions.

Residual-Aided End-to-End Learning of Communication System Without Known Channel

Leveraging powerful deep learning techniques, the end-to-end (E2E) learning of communication system is able to outperform the classical communication system. Unfortunately, this communication system cannot be trained by deep learning without known channel. To deal with this problem, a generative adversarial network (GAN) based training scheme has been recently proposed to imitate the real channel. However, the gradient vanishing and overfitting problems of GAN will result in the serious performance degradation of E2E learning of communication system. To mitigate these two problems, we propose a residual aided GAN (RA-GAN) based training scheme in this paper. Particularly, inspired by the idea of residual learning, we propose a residual generator to mitigate the gradient vanishing problem by realizing a more robust gradient backpropagation. Moreover, to cope with the overfitting problem, we reconstruct the loss function for training by adding a regularizer, which limits the representation ability of RA-GAN. Simulation results show that the trained residual generator has better generation performance than the conventional generator, and the proposed RA-GAN based training scheme can achieve the near-optimal block error rate (BLER) performance with a negligible computational complexity increase in both the theoretical channel model and the ray-tracing based channel dataset.

Adaptive-neuro-fuzzy-inference-system model for prediction of ultimate load of rectangular concrete-filled steel tubular columns

This study is devoted to the development of an Adaptive-Neuro-Fuzzy-Inference-System (ANFIS) model for the prediction of ultimate load of rectangular concrete-filled steel tubular structural members. The learning process of the model is performed by conducting a combination of backpropagation gradient descent and least-squares techniques. The performance of the model is examined by several quality metrics such as coefficient of determination (R2), Root-Mean-Squared-Error (RMSE), Mean-Absolute-Error (MAE), Index of Agreement (IA) and Slope of linear regression. Monte Carlo random sampling technique is employed to propagate input variations to the output response. Moreover, the performance of ANFIS is also compared with other machine learning models including Artificial Neural Network (ANN), Support Vector Machine (SVM), Gaussian Process Regression (GPR) and Ensemble. Results show that the ANFIS model yields higher prediction performance than other machine learning models, for both training and testing data points and regarding all quality metrics. For instance, using training data points, the ANFIS model exhibits a RMSE of 0.0283 compared to 0.0342, 0.0588, 0.0291, and 0.0464 using ANN, Ensemble, GPR, and SVM, respectively (the corresponding gain values are+17.3%,+51.9%,+2.8%, and+39.0%, respectively). On the other hand, using testing data points, the ANFIS model exhibits a RMSE of 0.0276 compared to 0.0393, 0.0987, 0.0403, and 0.0460 using ANN, Ensemble, GPR, and SVM, respectively (the corresponding values of gain are+29.8%,+72.1%,+31.5%, and+40.0%, respectively). The same observation can be made for other quality metrics. It can be concluded that the ANFIS model outperforms other models for both training and testing datasets. The ANFIS model is also compared with existing works in the past, showing its improvement in prediction results. Finally, sensitivity analysis is performed to determine the degree of effect of the input parameters on the ultimate load.

Automatic lung segmentation in chest X-ray images using improved U-Net

The automatic segmentation of the lung region for chest X-ray (CXR) can help doctors diagnose many lung diseases. However, extreme lung shape changes and fuzzy lung regions caused by serious lung diseases may incorrectly make the automatic lung segmentation model. We improved the U-Net network by using the pre-training Efficientnet-b4 as the encoder and the Residual block and the LeakyReLU activation function in the decoder. The network can extract Lung field features efficiently and avoid the gradient instability caused by the multiplication effect in gradient backpropagation. Compared with the traditional U-Net model, our method improves about 2.5% dice coefficient and 6% Jaccard Index for the two benchmark lung segmentation datasets. Our model improves about 5% dice coefficient and 9% Jaccard Index for the private lung segmentation datasets compared with the traditional U-Net model. Comparative experiments show that our method can improve the accuracy of lung segmentation of CXR images and it has a lower standard deviation and good robustness.

Effect of Hidden Neuron Size on Different Training Algorithm in Neural Network

We use different types of training algorithms in the neural network. But, we cannot say which kind of training algorithm is fast for a given problem. So, in this survey paper, we are trying to find which types of training algorithm are better for categorization problems. For this purpose, we used ten types of training algorithms in the pattern network in MATLAB. We used Levenberg-Marquardt (LM), Bayesian regularization backpropagation (BR), BFGS Quasi-Newton (BFG), Resilient Backpropagation (RP), Scaled Conjugate gradient backpropagation (SCG), Conjugate Gradient with Powell/Beale Restarts (CGB), Fletcher-Powell Conjugate Gradient (CGF), Polak-Ribiere Conjugate Gradient (GDM), One Step Secant (OSS), and Variable Learning Rate Backpropagation (GD) algorithm. In this survey paper, we also check, affects of these algorithms on neural network when we applied different types of hidden neuron size. During this survey we found some new facts. We found that RP, SCG, CGB, CGF and OSS are fastest algorithms. BFG takes more time with respect to hidden neuron size. GDM and GD take more epochs. BR algorithm is not acceptable for image categorization.

Neural stylist: Towards online styling service

Online stylist service enjoys huge economic potentials due to the trend of transformation of the fashion industry to digitalisation. Existing works either predict the fashion compatibility from the overall aspect or evaluate the compatibility with type-conditional representations. The prediction is hard to interpret due to the abstractive forecast. This paper proposes a visual and semantic representation model for explainable evaluation and recommendation. The model considers fashion compatibility from different factors, such as colour, material and style, by leveraging low to high-level features from former to later layers of CNN. The colour correlation and the pairwise relationship of fashion items in the same outfit are considered during the prediction stage. Instead of just predicting an outfit as compatible or incompatible, the model can classify an outfit as three precise evaluation levels: Good, Normal and Bad. The detailed compatible level is more consistent with the fashion sense of our human brain as Good or Bad outfits may have specific characteristics while Normal outfits tend to be ordinary. Additionally, the model can diagnose and recommend substitutions of the problematic fashion items from overall compatibility or colour-specific aspects by tracking the prediction matrices’ backpropagation gradients during the recommendation stage. Experiments in terms of outfit compatibility prediction and fill in the blank are conducted to evaluate the prediction ability of the proposed model. In contrast, fashion substitution recommendation experiments are conducted to assess the compatibility diagnosis and recommendation ability. Quantitative and qualitative results show that the model enables online stylist services with excellent explainability and generalisation on fashion prediction and recommendation.

A multi-layer spiking neural network-based approach to bearing fault diagnosis

Effective fault diagnosis is a crucial way to reduce the occurrence of severe damages of many industrial products. With the increasing amount of condition monitoring data, deep-learning-based methods have become promising ways for intelligent fault diagnosis thanks to their automatic feature extraction capability. Most recently, the third-generation neural network, called spiking neural network (SNN), has been introduced as an effective tool for fault diagnosis. However, the internal state and the error function of neurons in the SNN model cannot satisfy the conditions of continuity and differentiability, resulting in the difficulty of the gradient back-propagation, and it, therefore, prevents the extension of the SNN to a deep manner. In this article, a probabilistic spiking response model (PSRM) with a multi-layer structure is put forth to enhance the performance of the SNN in terms of bearing fault diagnosis. In the PSRM, the extracted features from the local mean decomposition (LMD) method are converted into the probability pulse sequences, and a multi-layer learning algorithm is developed to facilitate the multi-layer network training. The fault diagnosis results from three bearing databases, i.e., CWRU, MFPT, and Paderborn University datasets, demonstrate that the proposed PSRM exceeds a majority of the state-of-the-art machine learning methods. The proposed multi-layer SNN can also provide transparency to different bearing fault patterns by the membrane potentials of the spiking neurons in the output layer.

Neural network based autonomous control of a speech synthesis system

• Control of composite tasks by adaptive heuristically enhanced gradient approximation. • Training an artificial neural network (ANN) to control the parameters of a speech synthesizer. • The training error is estimated from the “quality” of the produced speech and not at the outputs of the ANN. • Currently used ANN training algorithms cannot cope with the above requirement. • The presented methodology may be applied to other applications too, e.g., coordination of robotic manipulators. This work is inspired by the ability of neural systems to control the behavior of specialized actuator mechanisms in living organisms by monitoring the end-effect of their actions. We consider as an example of such an actuator mechanism the human vocal tract where neurons learn to activate its muscles that move the velum, jaw, tongue and the lips, in order to exhibit desired phonetic activity. As a technical approximation to such a setup, we use an artificial neural network (ANN) and a speech synthesizer and we study the capability of the ANN to estimate the synthesizer’s parameters targeting desired speech activity. In this setup, we assume that the training error is obtained by measuring the “perceived distance” between the original (target) and the synthesized speech signals. Thus, the training error needs to be measured after processing the output of the speech synthesizer, instead of measuring it directly at the outputs of the ANN. This operational requirement on error measurement restricts the application of widely used ANN training algorithms that are based on back propagation of gradients but can be met by our earlier proposed “Heuristically Enhanced Gradient Approximation” (HEGA) algorithm. We also propose enhancements to HEGA that further optimize its performance in this demanding application.

Determining the Risky Software Projects using Artificial Neural Networks

Determining risky software projects early is a very important factor for project success. In this study it is aimed to choose the most correctly resulting modelling method that will be useful for early prediction of risky software projects to help companies to avoid losing time and money on unsuccessful projects and also facing legal requirements because of not being able to fullfill their responsibilites to their customers While making the research for this subject, it is seen that in previous researches, usually traditional modelling techniques were preferred. But it is observed that these methods were mostly resulted with high misclassification ratio. To overcome this problem, this study proposes a three-layered neural network (NN) architecture with a backpropagation algorithm. NN architecture was trained by using two different data sets which were OMRON data set (collected by OMRON) and 2016-2020 ES.LV data set (collected by the authors) separately. For the made of this study firstly the most relevant classification method (Gaussian Naive Bayes Algorithm) and the most relevant neural network method (Scaled Conjugate Gradient Backpropagation Algorithm) was chosen and both data sets were trained by using each method seperately for the purpose of observing which type of modelling architecture would give better results. Experimental results of this study showed that the neural network approach is useful for predicting whether a project is risky or not risky.

Stock Ranking with Multi-Task Learning

Stock prediction, aiming at predicting the future trends of stocks, plays a key role in stock investment. Towards the investment target, the primary task is selecting the stocks with potentials to obtain the highest excess returns, always regarded as stock ranking. List-wise stock ranking is able to consider the relative comparisons of multiple stocks, approaching the essence of stock ranking most. However, most existing methods fail in list-wise stock ranking, because the information complexity and small number of samples bring in training difficulties.To address these limitations, a novel Deep Multi-Task Learning (DMTL) solution is proposed, called Multi-Task Stock Ranking (MTSR). It utilizes the joint learning framework of DMTL to learn the list-wise stock ranking with the enhancements of auxiliary tasks. With DMTL, the easily-trained tasks act as learning guider, providing extra gradient backpropagation, to help learn the hardly-trained list-wise ranking task. Additionally, Task Relation Attention is utilized to capture the dynamic task relations to achieve better knowledge transfer between tasks. The experiments conducted on real-world stock datasets demonstrate the superiority of MTSR over several state-of-the-art methods.