Gradient Descent Training Research Articles

A TVD neural network closure and application to turbulent combustion

Trained neural networks (NN) have attractive features for closing governing equations. There are many methods that are showing promise, but all can fail in cases when small errors consequentially violate physical reality, such as a solution boundedness condition. A NN formulation is introduced to preclude spurious oscillations that violate solution boundedness or positivity. It is embedded in the discretized equations as a machine learning closure and strictly constrained, inspired by total variation diminishing (TVD) methods for hyperbolic conservation laws. The constraint is exactly enforced during gradient-descent training by rescaling the NN parameters, which maps them onto an explicit feasible set. Demonstrations show that the constrained NN closure model usefully recovers linear and nonlinear hyperbolic phenomena and anti-diffusion while enforcing the non-oscillatory property. Finally, the model is applied to subgrid-scale (SGS) modeling of a turbulent reacting flow, for which it suppresses spurious oscillations in scalar fields that otherwise violate the solution boundedness. It outperforms a simple penalization of oscillations in the loss function.

Reinforcement learning-based robust formation control for Multi-UAV systems with switching communication topologies

This paper introduces a novel optimal robust formation method for quadcopter multiple unmanned aerial vehicle (multi-UAV) systems. Firstly, a reinforcement learning (RL) algorithm based on a unique gradient descent training approach is proposed to solve the Hamilton–Jacobi–Bellman (HJB) equation, which can effectively eliminate the requirement of the Persistent Excitation (PE) condition. Secondly, the robustness of the controlled system is emphasized, and an Uncertainty and Disturbance Estimator (UDE) observer is developed to suppress model uncertainty and external disturbances through filtering techniques. Furthermore, a switched sliding mode control technique according to the average dwell time (ADT) is employed to convert switching communication topology between UAVs dynamically, and the stability analysis of the corresponding closed-loop control systems is then performed by the use of Lyapunov analysis. Finally, the simulation examples are provided to verify the effectiveness of the designed control strategy.

Exploring neural oscillations during speech perception via surrogate gradient spiking neural networks.

Understanding cognitive processes in the brain demands sophisticated models capable of replicating neural dynamics at large scales. We present a physiologically inspired speech recognition architecture, compatible and scalable with deep learning frameworks, and demonstrate that end-to-end gradient descent training leads to the emergence of neural oscillations in the central spiking neural network. Significant cross-frequency couplings, indicative of these oscillations, are measured within and across network layers during speech processing, whereas no such interactions are observed when handling background noise inputs. Furthermore, our findings highlight the crucial inhibitory role of feedback mechanisms, such as spike frequency adaptation and recurrent connections, in regulating and synchronizing neural activity to improve recognition performance. Overall, on top of developing our understanding of synchronization phenomena notably observed in the human auditory pathway, our architecture exhibits dynamic and efficient information processing, with relevance to neuromorphic technology.

Machine learning based analysis of entropy production and wall shear stress in blood flow through stenosed artery

The building up of cholesterol and various substances on the inner boundary of arterial walls forms a plaque (stenosis), which results in the narrowing of walls of arteries, and this exerts additional stress on the walls. Shear stress at the wall of arteries may cause the formation of plaque, plaque growth, and expansive arterial remodeling, and it may also result in an increased risk of plaque rupture. The purpose of this research work is to use machine learning approaches to analyze the wall shear stress and entropy generation through irreversible processes of blood flow in a porous stenosed artery within the context of magnetohydrodynamic consideration. As a way of illustrating blood’s non-Newtonian behavior, the Casson fluid model is used. A blood flow model in the context of mild stenosis has been developed, and it has subsequently been computationally addressed using an explicit finite difference method. The velocity, temperature, entropy generation, Bejan number, and wall shear stress are visualized for different values of pertinent variables. The result reflects the increased velocity in the stenosed region. A hybrid machine learning method using artificial neural network and particle swarm optimization is implemented to optimize the entropy generation and wall shear stress. The network is trained using Stochastic gradient–descent training procedure as well as Levenberg–Marquardt training procedure, and the outcomes of both training procedures are compared. Using this machine learning process, the minimum entropy production has been found for specific values of the Hartmann number, dimensionless temperature difference, and pulsatile component of pressure gradient. This machine learning-based optimization approach enhances the ability to forecast quantitative changes in affecting parameters, which will contribute to minimized damage to the arterial wall, improved blood flow efficiency and the delivery of necessary components to tissues.

Compact Model Training by Low-Rank Projection With Energy Transfer.

Low-rankness plays an important role in traditional machine learning but is not so popular in deep learning. Most previous low-rank network compression methods compress networks by approximating pretrained models and retraining. However, the optimal solution in the Euclidean space may be quite different from the one with low-rank constraint. A well-pretrained model is not a good initialization for the model with low-rank constraints. Thus, the performance of a low-rank compressed network degrades significantly. Compared with other network compression methods such as pruning, low-rank methods attract less attention in recent years. In this article, we devise a new training method, low-rank projection with energy transfer (LRPET), that trains low-rank compressed networks from scratch and achieves competitive performance. We propose to alternately perform stochastic gradient descent training and projection of each weight matrix onto the corresponding low-rank manifold. Compared to retraining on the compact model, this enables full utilization of model capacity since solution space is relaxed back to Euclidean space after projection. The matrix energy (the sum of squares of singular values) reduction caused by projection is compensated by energy transfer. We uniformly transfer the energy of the pruned singular values to the remaining ones. We theoretically show that energy transfer eases the trend of gradient vanishing caused by projection. In modern networks, a batch normalization (BN) layer can be merged into the previous convolution layer for inference, thereby influencing the optimal low-rank approximation (LRA) of the previous layer. We propose BN rectification to cut off its effect on the optimal LRA, which further improves the performance. Comprehensive experiments on CIFAR-10 and ImageNet have justified that our method is superior to other low-rank compression methods and also outperforms recent state-of-the-art pruning methods. For object detection and semantic segmentation, our method still achieves good compression results. In addition, we combine LRPET with quantization and hashing methods and achieve even better compression than the original single method. We further apply it in Transformer-based models to demonstrate its transferability. Our code is available at https://github.com/BZQLin/LRPET.

Multi-mode industrial soft sensor method based on mixture Laplace variational auto-encoder

The industrially collected process data usually exhibit non-Gaussian and multi-mode characteristics. Due to sensor failures, irregular disturbances, and transmission problems, there are unavoidable outliers that make the data exhibit heavy-tailed characteristics. To this end, a variational auto-encoder regression method based on the mixture Laplacian distribution (MLVAER) is proposed, by introducing a type-II multivariate Laplacian distribution in the latent variable space for robust modeling, and further extending it to the mixture form to accommodate multi-mode processes, the corresponding reparameterization trick is finally proposed for the mixture form of this distribution for neural network gradient descent training. The model based on this distribution assumption has higher degrees of freedom than the model based on the traditional multivariate Laplace distribution assumption when the network structure is the same. Numerical simulation and experiments on two industrial examples demonstrate that the proposed algorithm reduces the root mean square error by over 15% compared to other algorithms.

Controlled descent training

SummaryIn this work, a novel and model‐based artificial neural network (ANN) training method is developed supported by optimal control theory. The method augments training labels in order to robustly guarantee training loss convergence and improve training convergence rate. Dynamic label augmentation is proposed within the framework of gradient descent training where the convergence of training loss is controlled. First, we capture the training behavior with the help of empirical Neural Tangent Kernels (NTK) and borrow tools from systems and control theory to analyze both the local and global training dynamics (e.g., stability, reachability). Second, we propose to dynamically alter the gradient descent training mechanism via fictitious labels as control inputs and an optimal state feedback policy. In this way, we enforce locally optimal and convergent training behavior. The novel algorithm, Controlled Descent Training (CDT), guarantees local convergence. CDT unleashes new potentials in the analysis, interpretation, and design of ANN architectures. The applicability of the method is demonstrated on standard regression and classification problems.

Implicit Regularization of Dropout.

It is important to understand how dropout, a popular regularization method, aids in achieving a good generalization solution during neural network training. In this work, we present a theoretical derivation of an implicit regularization of dropout, which is validated by a series of experiments. Additionally, we numerically study two implications of the implicit regularization, which intuitively rationalizes why dropout helps generalization. First, we find that input weights of hidden neurons tend to condense on isolated orientations trained with dropout. Condensation is a feature in the non-linear learning process, which makes the network less complex. Second, we find that the training with dropout leads to the neural network with a flatter minimum compared with standard gradient descent training, and the implicit regularization is the key to finding flat solutions. Although our theory mainly focuses on dropout used in the last hidden layer, our experiments apply to general dropout in training neural networks. This work points out a distinct characteristic of dropout compared with stochastic gradient descent and serves as an important basis for fully understanding dropout.

Efficient stochastic parallel gradient descent training for on-chip optical processor

Efficient stochastic parallel gradient descent training for on-chip optical processor

Separable Gaussian Neural Networks: Structure, Analysis, and Function Approximations

The Gaussian-radial-basis function neural network (GRBFNN) has been a popular choice for interpolation and classification. However, it is computationally intensive when the dimension of the input vector is high. To address this issue, we propose a new feedforward network-separable Gaussian neural network (SGNN) by taking advantage of the separable property of Gaussian-radial-basis functions, which splits input data into multiple columns and sequentially feeds them into parallel layers formed by uni-variate Gaussian functions. This structure reduces the number of neurons from O(Nd) of GRBFNN to O(dN), which exponentially improves the computational speed of SGNN and makes it scale linearly as the input dimension increases. In addition, SGNN can preserve the dominant subspace of the Hessian matrix of GRBFNN in gradient descent training, leading to a similar level of accuracy to GRBFNN. It is experimentally demonstrated that SGNN can achieve an acceleration of 100 times with a similar level of accuracy over GRBFNN on tri-variate function approximations. The SGNN also has better trainability and is more tuning-friendly than DNNs with RuLU and Sigmoid functions. For approximating functions with a complex geometry, SGNN can lead to results that are three orders of magnitude more accurate than those of a RuLU-DNN with twice the number of layers and the number of neurons per layer.

Model order reduction methods applied to neural network training

AbstractNeural networks have emerged as powerful and versatile tools in the field of deep learning. As the complexity of the task increases, so do size and architectural complexity of the network, causing compression techniques to become a focus of current research. Parameter truncation can provide a significant reduction in memory and computational complexity. Originating from a model order reduction framework, the Discrete Empirical Interpolation Method is applied to the gradient descent training of neural networks and analyze for important parameters. The approach for various state‐of‐the‐art neural networks is compared to established truncation methods. Further metrics like L2 and Cross‐Entropy Loss, as well as accuracy and compression rate are reported.

Artificial Neural Network Model to Forecast Energy Consumption in Wheat Production in India

Energy analysis in agriculture sector require modelling technique that can incorporate complex unknown interactions and non-linearity in systems. In this study Artificial neural network technique is used to model and forecast input energy consumed in wheat production in India and is compared for predictive accuracy with linear models. Household data from 256 farmers revealed that the average input energy consumed in region is 29612.43 MJ/ha with urea (47%), diesel (31.5%) and electricity (9.8%) being three main contributors. Multi-layered feed forward model with 2 hidden layers with 8 and 15 neurons respectively and sigmoidal activation function in hidden layers and output layers under gradient descent training algorithm gave the best results. The R2 was 0.99 for training dataset and 0.973 for validation data set, while for MLR model it was 0.95 and 0.73 for respective datasets. The root mean squared error (RMSE) in ANN model was 4779.2 MJ/ha and 2008.96 MJ/ha for training and validation data, respectively. This prediction system could forecast input energy with error margin of ± 7889.83 MJ/ha on training dataset and ± 3298.47 MJ/ha on validation data under various combinations. Sensitivity analysis showed that urea, diesel, and electricity are the important factors in input energy forecasting.

MOSAL: A Subspace-Based Forecasting Algorithm for Throughput Maximization in IoT Networks

Predictive solution techniques have been developed recently to solve the massive access problem of the Internet of Things (IoT). These techniques forecast the traffic generation patterns of individual IoT devices in the coverage area of an IoT gateway and schedule the Medium Access Control (MAC)-layer resources at the gateway in advance based on these forecasts. Although predictive solutions have achieved high network performance, a key problem is that their performance depends highly on the performance of forecasters. In this article, to minimize the effects of forecasting errors on the performance of predictive networks, we develop a subspace-based forecasting algorithm called “Motion On a Subspace under Adaptive Learning rate (MOSAL).” First, our algorithm trains a forecaster by minimizing the performance loss of an IoT network based on the emulation of an Application-Specific Error Function (ASEF) by an Artificial Neural Network (ANN). Second, the algorithm moves close to a subspace of the forecasting errors while aiming to maximize network throughput. Our results show that MOSAL achieves a throughput performance that surpasses the performance of commonly used standard gradient descent training algorithms at a reasonable execution time. These results open the way to the deployment of predictive solutions at IoT gateways in practice in the near future.

Initial Work on the Development of a Hardware-Based Gradient Descent Trained Expert System

Prior work has introduced a form of explainable artificial intelligence that is able to precisely explain, in a human-understandable form, why it makes decisions. It is also able to learn to make better decisions without potentially learning illegal or invalid considerations. This defensible system is based on fractional value rule-fact expert systems and the use of gradient descent training to optimize rule weightings. This software system has demonstrated efficacy for many applications; however, it utilizes iterative processing and thus does not have a deterministic completion time. It also requires comparatively expensive general-purpose computing hardware to run on. This paper builds on prior work in the development of hardware-based expert systems and presents and assesses the efficacy of a hardware implementation of this system. It characterizes its performance and discusses its utility and trade-offs for several application domains.

Model predictive control for multi-zone Variable Air Volume systems based on artificial neural networks

In this paper, an artificial neural network (ANN) based system-level model predictive control framework is established for a variable air volume (VAV) system to improve its robustness and energy efficiency. The VAV system consists of three processes: the zone temperature process, the damper process and the supply air volume process of the air handling unit connecting the ductwork for all of the boxes. An ANN-based predictive controller is designed for the zone temperature process, and a PI controller is used for the damper process; they operate in a cascaded manner. Another ANN predictive controller tracks the total supply air volume subject to the cooling load constraints from the lower level VAV boxes and minimizes the fan energy consumption. The ANN controllers are optimized online by the Lagrange variational approach (LVA) to minimize the system cost function and implement a feedback receding horizon optimal control. With the LVA based gradient descent training algorithm, the weight convergence of the ANN predictive controllers and stability of the control system is obtained with small computational and storage needs. The modeling methodology, based on ANNs for the above processes, is developed and calibrated with actual data measured from a site. All of the data processing and control strategy implementations are based on extensive measurements collected from a laboratory building located in the campus of Beijing University of Civil Engineering and Architecture, Beijing, China. The experimental results show that an additional 6.12% energy savings is achieved. This is accomplished by introducing the fan control signal into the system cost function as the performance index of energy consumption with a 0.2 penalty coefficient. In the presence of un-modeled disturbances and modeling errors, the framework also reduces the temperature oscillation at least by approximately 60% compared with PI control.

Artificial Intelligence Scheme for Medical Images Classification and Prediction

Medicine is the industry where smart technologies and artificial intelligence are most commonly used. Medical imaging is usually used for tumor diagnosis; this includes Computer Tomography and Magnetic Resonance Imaging. Early tumor detection in various organs based on such images is important. This study intended to present an Adaptive Convolution Neural Networks (ACNN) based method for tumor detection in the brain. The ACNN will utilize a modified stochastic gradient descent (MSGD) training algorithm with adaptive momentum and learning rates to speed up the convergence of the error, which will speed up the classification process and improve the accuracy. MSGD is implemented such as when the loss increases, the learning rate increase, and vice versa. The proposed modifications allow the network to increase the learning rate at the beginning of the training process and slow down as the network outcomes reach stabilized conditions. The proposed method results were compared against the performance of several conventional combinations of CNN with several machine learning classifiers. The test results show that the proposed method outperformed the performance of the CNN with all the above-said adaptations. Accordingly, the contributions of this study are (1) improving the ACNN training algorithm for the tumor classification problem and (2) proposing original CNN architectures specialized for tumor classification. Keywords: artificial intelligence; convolution neural network; medical images processing; tumor classification, tumor detection.

Automated Machine Learning (AutoML)-Derived Preconception Predictive Risk Model to Guide Early Intervention for Gestational Diabetes Mellitus.

The increasing prevalence of gestational diabetes mellitus (GDM) is contributing to the rising global burden of type 2 diabetes (T2D) and intergenerational cycle of chronic metabolic disorders. Primary lifestyle interventions to manage GDM, including second trimester dietary and exercise guidance, have met with limited success due to late implementation, poor adherence and generic guidelines. In this study, we aimed to build a preconception-based GDM predictor to enable early intervention. We also assessed the associations of top predictors with GDM and adverse birth outcomes. Our evolutionary algorithm-based automated machine learning (AutoML) model was implemented with data from 222 Asian multi-ethnic women in a preconception cohort study, Singapore Preconception Study of Long-Term Maternal and Child Outcomes (S-PRESTO). A stacked ensemble model with a gradient boosting classifier and linear support vector machine classifier (stochastic gradient descent training) was derived using genetic programming, achieving an excellent AUC of 0.93 based on four features (glycated hemoglobin A1c (HbA1c), mean arterial blood pressure, fasting insulin, triglycerides/HDL ratio). The results of multivariate logistic regression model showed that each 1 mmol/mol increase in preconception HbA1c was positively associated with increased risks of GDM (p = 0.001, odds ratio (95% CI) 1.34 (1.13–1.60)) and preterm birth (p = 0.011, odds ratio 1.63 (1.12–2.38)). Optimal control of preconception HbA1c may aid in preventing GDM and reducing the incidence of preterm birth. Our trained predictor has been deployed as a web application that can be easily employed in GDM intervention programs, prior to conception.

Neuromorphic Implementation of Logic Functions Based on Parallel Dual-Gate Thin-Film Transistors

Composed of computation units each containing a parallel, dual-gate (DG) thin-film transistor (TFT) and a memory capacitor addressed by an access TFT, a neuromorphic circuit implementing fundamental AND and OR logic functions is realized and characterized. The exceptionally low leakage current of a TFT built on indium-gallium-zinc oxide (IGZO) allows the capacitor to be used as a quasi-static memory element. Appropriate weight signals for the realization of the two logic functions are obtained using a gradient-descent training algorithm. The circuit can be configured to execute one of the two functions by the setting of the weight signals.

Accuracy and Resiliency of Analog Compute-in-Memory Inference Engines

Recently, analog compute-in-memory (CIM) architectures based on emerging analog non-volatile memory (NVM) technologies have been explored for deep neural networks (DNNs) to improve scalability, speed, and energy efficiency. Such architectures, however, leverage charge conservation, an operation with infinite resolution, and thus are susceptible to errors. Thus, the inherent stochasticity in any analog NVM used to execute DNNs, will compromise performance. Several reports have demonstrated the use of analog NVM for CIM in a limited scale. It is unclear whether the uncertainties in computations will prohibit large-scale DNNs. To explore this critical issue of scalability, this article first presents a simulation framework to evaluate the feasibility of large-scale DNNs based on CIM architecture and analog NVM. Simulation results show that DNNs trained for high-precision digital computing engines are not resilient against the uncertainty of the analog NVM devices. To avoid such catastrophic failures, this article introduces the analog bi-scale representation for the DNN, and the Hessian-aware Stochastic Gradient Descent training algorithm to enhance the inference accuracy of trained DNNs. As a result of such enhancements, DNNs such as Wide ResNets for CIFAR-100 image recognition problem are demonstrated to have significant performance improvements in accuracy without adding cost to the inference hardware .

Universality of gradient descent neural network training

It has been observed that design choices of neural networks are often crucial for their successful optimization. In this article, we therefore discuss the question if it is always possible to redesign a neural network so that it trains well with gradient descent. This yields the following universality result: If, for a given network, there is any algorithm that can find good network weights for a classification task, then there exists an extension of this network that reproduces the same forward model by mere gradient descent training. The construction is not intended for practical computations, but it provides some orientation on the possibilities of pre-trained networks in meta-learning and related approaches.