Forward-backward Propagation Research Articles

XGrad: Boosting Gradient-Based Optimizers With Weight Prediction.

In this paper, we propose a general deep learning training framework XGrad which introduces weight prediction into the popular gradient-based optimizers to boost their convergence and generalization when training the deep neural network (DNN) models. In particular, ahead of each mini-batch training, the future weights are predicted according to the update rule of the used optimizer and are then applied to both the forward pass and backward propagation. In this way, during the whole training period, the optimizer always utilizes the gradients w.r.t. the future weights to update the DNN parameters, making the gradient-based optimizer achieve better convergence and generalization compared to the original optimizer without weight prediction. XGrad is rather straightforward to implement yet pretty effective in boosting the convergence of gradient-based optimizers and the accuracy of DNN models. Empirical results concerning five popular optimizers including SGD with momentum, Adam, AdamW, AdaBelief, and AdaM3 demonstrate the effectiveness of our proposal. The experimental results validate that XGrad can attain higher model accuracy than the baseline optimizers when training the DNN models.

Quantitative Convergence for Displacement Monotone Mean Field Games with Controlled Volatility

We study the convergence problem for mean field games with common noise and controlled volatility. We adopt the strategy recently put forth by Laurière and the second author, using the maximum principle to recast the convergence problem as a question of “forward-backward propagation of chaos” (i.e., (conditional) propagation of chaos for systems of particles evolving forward and backward in time). Our main results show that displacement monotonicity can be used to obtain this propagation of chaos, which leads to quantitative convergence results for open-loop Nash equilibria for a class of mean field games. Our results seem to be the first (quantitative or qualitative) that apply to games in which the common noise is controlled. The proofs are relatively simple and rely on a well-known technique for proving wellposedness of forward-backward stochastic differential equations, which is combined with displacement monotonicity in a novel way. To demonstrate the flexibility of the approach, we also use the same arguments to obtain convergence results for a class of infinite horizon discounted mean field games. Funding: J. Jackson is supported by the National Science Foundation [Grant DGE1610403]. L. Tangpi is partially supported by the National Science Foundation [Grants DMS-2005832 and DMS-2143861].

A study on the prediction of development strategies of art education in universities based on deep learning model

Abstract In this paper, we propose a deep learning restoration method for fine art painting images based on a contextual encoder and select a convolutional self-encoder-based image restoration model for workflow analysis, including forward inference and backward network propagation. We design and develop a software system for predicting the grades of college art and painting students and apply the neural network prediction model to this system to realize the grade prediction of college art students. The accuracy of the GDPN model is 0.922, the precision is 0.904, and the recall is 0.966, which can consider both the temporal and overall information in the click behavior and achieve a better prediction effect.

A Multilayer Framework for Online Metric Learning.

Online metric learning (OML) has been widely applied in classification and retrieval. It can automatically learn a suitable metric from data by restricting similar instances to be separated from dissimilar instances with a given margin. However, the existing OML algorithms have limited performance in real-world classifications, especially, when data distributions are complex. To this end, this article proposes a multilayer framework for OML to capture the nonlinear similarities among instances. Different from the traditional OML, which can only learn one metric space, the proposed multilayer OML (MLOML) takes an OML algorithm as a metric layer and learns multiple hierarchical metric spaces, where each metric layer follows a nonlinear layer for the complicated data distribution. Moreover, the forward propagation (FP) strategy and backward propagation (BP) strategy are employed to train the hierarchical metric layers. To build a metric layer of the proposed MLOML, a new Mahalanobis-based OML (MOML) algorithm is presented based on the passive-aggressive strategy and one-pass triplet construction strategy. Furthermore, in a progressively and nonlinearly learning way, MLOML has a stronger learning ability than traditional OML in the case of limited available training data. To make the learning process more explainable and theoretically guaranteed, theoretical analysis is provided. The proposed MLOML enjoys several nice properties, indeed learns a metric progressively, and performs better on the benchmark datasets. Extensive experiments with different settings have been conducted to verify these properties of the proposed MLOML.

A 28-nm 8-bit Floating-Point Tensor Core-Based Programmable CNN Training Processor With Dynamic Structured Sparsity

Training deep/convolutional neural networks (DNNs/CNNs) requires a large amount of memory and iterative computation, which necessitates speedup and energy reduction, especially for edge devices with resource/energy constraints. In this work, we present an 8-bit floating-point (FP8) training the processor which implements: 1) highly parallel tensor cores (fused multiply–add trees) that maintain high utilization throughout forward propagation (FP), backward propagation (BP), and weight update (WU) phases of the training process; 2) hardware-efficient channel gating for dynamic output activation sparsity; 3) dynamic weight sparsity (WS) based on group Lasso; and 4) gradient skipping based on the FP prediction error. We develop a custom instruction set architecture (ISA) to flexibly support different CNN topologies and training parameters. The 28-nm prototype chip demonstrates large improvements in floating-point operations (FLOPs) reduction (7.3 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\times$</tex-math> </inline-formula> ), energy efficiency (16.4 TFLOPS/W), and overall training latency speedup (4.7 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\times$</tex-math> </inline-formula> ), for both supervised and self-supervised training tasks.

Multi-Depth Computer-Generated Hologram Based on Stochastic Gradient Descent Algorithm with Weighted Complex Loss Function and Masked Diffraction

In this paper, we propose a method to generate multi-depth phase-only holograms using stochastic gradient descent (SGD) algorithm with weighted complex loss function and masked multi-layer diffraction. The 3D scene can be represented by a combination of layers in different depths. In the wave propagation procedure of multiple layers in different depths, the complex amplitude of layers in different depths will gradually diffuse and produce occlusion at another layer. To solve this occlusion problem, a mask is used in the process of layers diffracting. Whether it is forward wave propagation or backward wave propagation of layers, the mask can reduce the occlusion problem between different layers. Otherwise, weighted complex loss function is implemented in the gradient descent optimization process, which analyzes the real part, the imaginary part, and the amplitude part of the focus region between the reconstructed images of the hologram and the target images. The weight parameter is used to adjust the ratio of the amplitude loss of the focus region in the whole loss function. The weight amplitude loss part in weighted complex loss function can decrease the interference of the focus region from the defocus region. The simulations and experiments have validated the effectiveness of the proposed method.

A bidirectional DNN partition mechanism for efficient pipeline parallel training in cloud

Recently, deep neural networks (DNNs) have shown great promise in many fields while their parameter sizes are rapidly expanding. To break through the computation and memory limitation of a single machine, pipeline model parallelism is proposed for large-scale DNN training by fully utilizing the computation and storage power of the distributed cluster. Cloud data centers can also provide sufficient computing, storage and bandwidth resources. However, most existing approaches apply layer-wise partitioning, which is difficult to obtain an even model partition result because of the large computational overhead discrepancy between DNN layers, resulting in degraded efficiency. To tackle this issue, we propose “Bi-Partition”, a novel partitioning method based on bidirectional partitioning for forward propagation (FP) and backward propagation (BP), which improves the efficiency of the pipeline model parallelism system. By deliberated designing distinct cut positions for FP and BP of DNN training, workers in the pipeline get nearly equal computational loads, and the balanced pipeline fully utilizes the computing resources. Experiments on various DNN models and datasets validate the efficiency of our mechanism, e.g., the training efficiency achieving up to 1.9times faster than the state-of-the-art method PipeDream.

Prior Distribution Refinement for Reference Trajectory Estimation With the Monte Carlo-Based Localization Algorithm

A robot localization problem demands a fair comparison of the positioning algorithms. A reference trajectory of the robot’s movement is needed to estimate errors and evaluate a quality of the localization. In this article, we propose the Prior Distribution Refinement method for generating a reference trajectory of a mobile robot with the Monte Carlo-based localization system. The proposed approach can be applied for both indoor and outdoor environments of an arbitrary size without the need for expensive position tracking sensors or intervention in the testing infrastructure. The reference trajectory is generated by running the algorithm over a so-called Particles’ Transition Graph, obtained from a resampling stage of Monte Carlo localization. The prior distribution of particles is then refined by forward-backward propagation through the graph and exploring the connections between particles. The Viterbi algorithm is applied afterwards to generate a reference trajectory based on refined particles’ distribution. We demonstrate that such an approach is capable of generating accurate estimates of a mobile robot’s position and orientation with the only requirement of moderate quality of localization system being used as a core algorithm for iterative optimization.

An On-Chip Fully Connected Neural Network Training Hardware Accelerator Based on Brain Float Point and Sparsity Awareness

In recent years, deep neural networks (DNNs) have brought revolutionary progress in various fields with the advent of technology. It is widely used in image pre-processing, image enhancement technology, face recognition, voice recognition, and other applications, gradually replacing traditional algorithms. It shows that the rise of neural networks has led to the reform of artificial intelligence. Since neural network algorithms are computationally intensive, they require GPUs or accelerated hardware for real-time computation. However, the high cost and high power consumption of GPUs result in low energy efficiency. It recently led to much research on accelerated digital circuit hardware design for deep neural networks. In this paper, we propose an efficient and flexible neural network training processor for fully connected layers. Our proposed training processor features low power consumption, high throughput, and high energy efficiency. It uses the sparsity of neuron activations to reduce the number of memory accesses and memory space to achieve an efficient training accelerator. The proposed processor uses a novel reconfigurable computing architecture to maintain high performance when operating Forward Propagation and Backward Propagation. The processor is implemented in Xilinx Zynq UltraSacle+MPSoC ZCU104 FPGA, with an operating frequency of 200MHz and power consumption of 6.444W, and can achieve 102.43 GOPS.

An optical-cryptographic encryption scheme using a keystream synthesizer based on chaotic speckle nature

We propose a comprehensive and efficient optical-cryptographic encryption scheme to improve the security of vulnerable optical encryption systems. Based on cryptography theory, we increase data protection using diffusion-confusion processes acting in the input and output of the optical encryption scheme. The new enhanced ciphertext has attributes of a one-way function that is easy to achieve but almost impossible to revert without the correct random phase masks used for diffusion-confusion operations. Despite the straightforward design, combining optical, cryptographic, and chaos methods increases the number of encryption–decryption keys raising two significant concerns. First, the management and transmission of several keys carry a high storage footprint, and second, it impairs the optical high-speed information processing capability. We overcome these two drawbacks by applying a new optical keystream synthesizer based on the stochastic speckle nature and a maximum entropy constraint in a forward–backward propagation algorithm. This new approach quickly replicates the necessary keystream in the encryption–decryption stations offering higher security and efficiency than other reported systems. We validate the feasibility of this proposal by synthesizing an experimental keystream and performing 24-bit data encryption–decryption. Then, we verify the security improvement of two vulnerable cryptosystems and their robustness against cryptanalysis attacks. Finally, we perform experimental encryption–decryption.

The Dual-Fuzzy Convolutional Neural Network to Deal With Handwritten Image Recognition

Subjective evaluation is a commonly used method in the real recognition process. Generally, two fuzziness can be found in evaluation information, namely what values should be given to fully describe the information and how to distinguish different values. The recently developed probabilistic hesitant fuzzy set could perfectly address these issues. In this article, we propose a dual-fuzzy convolutional neural network (DF-CNN) by fusing the hot neural network algorithm into the probabilistic hesitant fuzzy environment and then using it in a practical handwritten image recognition process. For this new DF-CNN, we provide the whole calculation process including the forward propagation, backward propagation, and parameter updating calculations. Also, the optimization algorithm of the DF-CNN is given to derive its optimal results. Finally, we apply the DF-CNN and its optimization algorithm to deal with a real issue, namely the handwritten numeral image recognition. The calculation process and the comparison fully demonstrate the feasibility and effectiveness of the proposed new model and algorithm.

Trajectory Recovery Based on Interval Forward–Backward Propagation Algorithm Fusing Multi-Source Information

In the tracking scheme in which global navigation satellite system (GNSS) measurement is temporally lost or the sampling frequency is insufficient, dead reckoning based on the inertial measurement unit (IMU) and other location-related information can be fused as a supplement for real-time trajectory recovery. The tracking scheme based on interval analysis outputs interval results containing the ground truth, which gives it the advantage of convenience in multi-source information fusion. In this paper, a trajectory-recovery algorithm based on interval analysis is proposed, which can conveniently fuse GNSS measurement, IMU data, and map constraints and then output an interval result containing the actual trajectory. In essence, the location-related information such as satellite measurement, inertial data, and map constraints is collected by practical experiments and then converted into interval form. Thereby, the interval-overlapping calculation is performed through forward and backward propagation to accomplish the trajectory recovery. The practical experimental results show that the trajectory recovery accuracy based on the proposed algorithm performs better than the traditional Kalman filter algorithm, and the estimated interval results deterministically contain the actual trajectory. More importantly, the proposed interval algorithm is approved to be convenient to fuse additional location-related information.

An Artificial Intelligence Approach to Manage Crop Water Requirements in South Africa

Estimation of crop water requirements is of paramount importance towards the management of agricultural water resources, which is a major mitigating strategy against the effects of climate change on food security. South Africa water shortage poses a threat on agricultural efficiency. Since irrigation uses about 60% of the fresh water available, it therefore becomes important to optimise the use of irrigation water in order to maximize crop yield at the farm level in order to avoid wastage. In this study, combined application of an artificial neural network (ANN) and a crop – growth simulation model for the estimation of crop irrigation water requirements and the irrigation scheduling of potatoes at Winterton irrigation scheme, South Africa was investigated. The crop-water demand from planting to harvest date, when to irrigate, the optimum stage in the drying cycle when to apply water and the amount of irrigation water to be applied per time, were estimated in this study. Five feed –forward backward propagation artificial neural network predictive models were developed with varied number of neurons and hidden layers and evaluated. The optimal ANN model, which has 5 inputs, 5 neurons, 1 hidden layer and 1 output was used to predict monthly reference evapotranspiration (ETo) in the Winterton area. The optimal ANN model produced a root-mean-square error (RMSE) of 0.67, Pearson correlation coefficient (r) of 0.97 and coefficient of determination (R2) of 0.94. The validation of the model between the measured and predicted ETo shows a r value of 0.9048. The predicted ETo was one of the input variables into a crop growth simulation model, called CROPWAT. The results indicated that the total crop water requirement was 1259.2 mm/decade and net irrigation water requirement was 1276.9 mm/decade, spread over a 5-day irrigation time during the entire 140 days of cropping season for potatoes. A combination of the artificial neural networks and the crop growth simulation models have proved to be a robust technique for estimating crop irrigation water requirements in the face of limited or no daily meteorological datasets.

Analysis of the Importance and Influence of Student Mental Health Based on Neural Networks

With the rapid development of neural network has been widely used in major research areas, this paper will neural network convolution layer structure into the importance of students' mental health and influencing factors. In order to analyze the influencing factors of students' mental health, this paper analyzes the mental health status of different students from the perspective of neural network. Firstly, the main definitions and concepts of students' mental health are put forward, and the methods of evaluating and measuring mental health are analyzed. Secondly, the related structure of neural network and two different neural network model algorithms are described in detail, and the forward propagation and backward propagation algorithms of neural network are proposed (which provide support for the data research through neural network later). Finally, the correlation function of neural network is used to analyze the influencing factors of contemporary students' mental health.

Improved Gradient-Based Adversarial Attacks for Quantized Networks

Neural network quantization has become increasingly popular due to efficient memory consumption and faster computation resulting from bitwise operations on the quantized networks. Even though they exhibit excellent generalization capabilities, their robustness properties are not well-understood. In this work, we systematically study the robustness of quantized networks against gradient based adversarial attacks and demonstrate that these quantized models suffer from gradient vanishing issues and show a fake sense of robustness. By attributing gradient vanishing to poor forward-backward signal propagation in the trained network, we introduce a simple temperature scaling approach to mitigate this issue while preserving the decision boundary. Despite being a simple modification to existing gradient based adversarial attacks, experiments on multiple image classification datasets with multiple network architectures demonstrate that our temperature scaled attacks obtain near-perfect success rate on quantized networks while outperforming original attacks on adversarially trained models as well as floating-point networks.

Evaluation of Sigmoid and ReLU Activation Functions Using Asymptotic Method

The performance of two algorithms may be compared using an asymptotic technique in algorithm analysis, which focuses largely on the growth rate as the number of inputs grows. Sigmoid activation and ReLU activation functions are widely employed in ANNs (Yingying, 2020), and each has advantages and disadvantages that should be considered when designing ANN solutions for a given issue. This study aimed to compare the performance of sigmoid activation and ReLU activation function during training using an asymptotic approach. The work focuses on training time complexity as the basis of comparison of the two activation functions using an asymptotic approach. The result derived from this study showed that sigmoid activation function takes more computation time in performing forward path, loss computation and backward propagation than ReLU activation functions. The computation cost will become significant when dealing with deep neural networks with hundreds to thousands of neurons. Overall, the training time for ReLU based Neural network will be better than that of sigmoid based one. Sigmoid have higher computational cost compared to ReLU but the two algorithms have a linear growth rate. Keyword: Back propagation, Loss computation, Sigmoid Activation, ReLU activation, ANNs DOI: 10.7176/NCS/13-05 Publication date: June 30th 2022

Design of Processing-in-Memory With Triple Computational Path and Sparsity Handling for Energy-Efficient DNN Training

As machine learning (ML) and artificial intelligence (AI) have become mainstream technologies, many accelerators have been proposed to cope with their computation kernels. However, they access the external memory frequently due to the large size of deep neural network model, suffering from the von Neumann bottleneck. Moreover, as privacy issue is becoming more critical, on-device training is emerging as its solution. However, on-device training is challenging because it should perform the training under a limited power budget, which requires a lot more computations and memory accesses than the inference. In this paper, we present an energy-efficient processing-in-memory (PIM) architecture supporting end-to-end on-device training named T-PIM. Its macro design includes an 8T-SRAM cell-based PIM block to compute in-memory <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">AND</i> operation and three computational datapaths for end-to-end training. Each of three computational paths integrates arithmetic units for forward propagation, backward propagation, and gradient calculation and weight update, respectively, allowing the weight data stored in the memory stationary. T-PIM also supports variable bit precision to cover various ML scenarios. It can use fully variable input bit precision and 2-bit, 4-bit, 8-bit, and 16-bit weight bit precision for the forward propagation and the same input bit precision and 16-bit weight bit precision for the backward propagation. In addition, T-PIM implements sparsity handling schemes that skip the computation for input data and turn off the arithmetic units for weight data to reduce both unnecessary computations and leakage power. Finally, we fabricate the T-PIM chip on a 5.04mm <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> die in a 28-nm CMOS logic process. It operates at 50–280MHz with the supply voltage of 0.75–1.05V, dissipating 5.25–51.23mW power in inference and 6.10–37.75mW in training. As a result, it achieves 17.90–161.08TOPS/W energy efficiency for the inference of 1-bit activation and 2-bit weight data, and 0.84–7.59TOPS/W for the training of 8-bit activation/error and 16-bit weight data. In conclusion, T-PIM is the first PIM chip that supports end-to-end training, demonstrating 2.02 times performance improvement over the latest PIM that partially supports training.

Training optronic convolutional neural networks on an optical system through backpropagation algorithms.

The development of optical neural networks greatly slows the urgent demand of searching for fast computing approaches to solve big data processing. However, most optical neural networks following electronic training and optical inferencing do not really take full advantage of optical computing to reduce computational burden. Take the extensively used optronic convolutional neural networks (OPCNN) as an example, the convolutional operations still require vast computational operations in training stages on the computer. To address this issue, this study proposes the in-situ training algorithm to train the networks directly in optics. We derive the backpropagation algorithms of OPCNN hence the complicated gradient calculation in backward propagating processes can be obtained through optical computing. Both forward propagation and backward propagation are all executed on the same optical system. Furthermore, we successfully realize the introduction of optical nonlinearity in networks through utilizing photorefractive crystal SBN:60 and we also derive the corresponding backpropagation algorithm. The numerical simulation results of classification performance on several datasets validates the feasibility of the proposed algorithms. Through in-situ training, the reduction in performance resulting from the inconsistency of the plantform between training and inferencing stages can be eliminated completely. For example, we demonstrate that by using the optical training approach, OPCNN is capable of gaining a strong robustness under several misalignmed situations, which enhances the practicability of OPCNN and greatly expands its application range.

Batch covariance neural network for image recognition

Recent work has shown that convolutional neural networks (CNN) can achieve state of the art if the datasets are well built. However, the existing convolutional layer is affected by various datasets with the inevitable problems of local abnormal features, i.e., illumination intensity and feature interaction. This paper replaces the convolutional layer with a batch covariance layer (BCL) to locate the category-related region unaffected by the problems. The BCL is regarded as a 3D covariance operation, which calculates the correlation between the kernels and feature maps in kernel size of all channels. Forward propagation, backward propagation, gradient updating, and testing procedure of the BCL are described. The comparison between BCL and convolutional layer shows the ability of BCL to reduce the influence of illumination intensity and feature interaction for discriminating and generating tasks. Complexity analysis shows that BCL can improve the accuracy with a thimbleful time consumption increase. Besides, the batch covariance neural network (BCovNN) is extended from the CNN by replacing the convolutional layer with BCL. Ablation experiment verifies the improvement of BCovNN is provided by BCL separately. BCovNN is evaluated on several popular datasets (i.e., MNIST, STL-10, CIFAR-10, and ImageNet) for image recognition and PASCAL VOC (2007 and 2012) datasets for object localization. Experimental results reveal that BCovNN achieves significant improvements over the corresponding CNN.

Methodology for the Assessment of Imprecise Multi-State System Availability

Most existing studies of a system’s availability in the presence of epistemic uncertainties assume that the system is binary. In this paper, a new methodology for the estimation of the availability of multi-state systems is developed, taking into consideration epistemic uncertainties. This paper formulates a combined approach, based on continuous Markov chains and interval contraction methods, to address the problem of computing the availability of multi-state systems with imprecise failure and repair rates. The interval constraint propagation method, which we refer to as the forward–backward propagation (FBP) contraction method, allows us to contract the probability intervals, keeping all the values that may be consistent with the set of constraints. This methodology is guaranteed, and several numerical examples of systems with complex architectures are studied.