Vanishing Gradient Research Articles

Cattle Body Intelligent Measurement based on Improved CenterNet

Traditional cattle body measurement methods have limitations such as manual contact measurement and low efficiency. To improve the efficiency of measuring cattle bodies and reduce labor costs, an improved intelligent measurement network based on CenterNet was proposed. In this study, we used the DenseNet-100 to replace the ResNet-101 to alleviate the vanishing gradient problem. We also improved the cattle body size measurement efficiency by reducing the network parameters. In addition, we conducted a comparative experiment on the cattle posture dataset collected by ourselves to verify the feasibility of the network. The experiment confirmed that the proposed intelligent measurement based on improved CenterNet outperforms the traditional networks and other advanced object measurement networks in accuracy and measurement efficiency, and can effectively solve most of the error problems caused by manual measurement. Besides, our network has good applicability and strong stability, which can meet the requirements of the evaluation of cattle body size measurement indicators.

Rail surface defect data enhancement method based on improved ACGAN.

Rail surface defects present a significant safety concern in railway operations. However, the scarcity of data poses challenges for employing deep learning in defect detection. This study proposes an enhanced ACGAN augmentation method to address these issues. Residual blocks mitigate vanishing gradient problems, while a spectral norm regularization-constrained discriminator improves stability and image quality. Substituting the generator's deconvolution layer with upsampling and convolution operations enhances computational efficiency. A gradient penalty mechanism based on regret values addresses gradient abnormality concerns. Experimental validation demonstrates superior image clarity and classification accuracy compared to ACGAN, with a 17.6% reduction in FID value. MNIST dataset experiments verify the model's generalization ability. This approach offers practical value for real-world applications.

A study on expression recognition based on improved mobilenetV2 network

This paper proposes an improved strategy for the MobileNetV2 neural network(I-MobileNetV2) in response to problems such as large parameter quantities in existing deep convolutional neural networks and the shortcomings of the lightweight neural network MobileNetV2 such as easy loss of feature information, poor real-time performance, and low accuracy rate in facial emotion recognition tasks. The network inherits the characteristics of MobilenetV2 depthwise separated convolution, signifying a reduction in computational load while maintaining a lightweight profile. It utilizes a reverse fusion mechanism to retain negative features, which makes the information less likely to be lost. The SELU activation function is used to replace the RELU6 activation function to avoid gradient vanishing. Meanwhile, to improve the feature recognition capability, the channel attention mechanism (Squeeze-and-Excitation Networks (SE-Net)) is integrated into the MobilenetV2 network. Experiments conducted on the facial expression datasets FER2013 and CK + showed that the proposed network model achieved facial expression recognition accuracies of 68.62% and 95.96%, improving upon the MobileNetV2 model by 0.72% and 6.14% respectively, and the parameter count decreased by 83.8%. These results empirically verify the effectiveness of the improvements made to the network model.

Hybrid hyperplane gradient learning algorithm for RBF neural network

Gradient-based algorithms are still popularly used for training radial basis function neural network (RBFNN). However, these algorithms may lead to the vanishing gradient (saddle point or local minimum) problem, which is one of the common problems that limit learning performance. To cope with this, a hybrid hyperplane gradient learning algorithm (HHGLA) is proposed to improve the learning performance of the RBFNN in this paper. First, a hyperplane gradient (HPG), based on a hyperspace constructed by a solution population (SP), is introduced. Thus, the search process can cross the surface of the cost function to avoid the appearance of vanishing gradient. Second, an adaptive learning rate is designed to restrict the search process into this hyperspace. Then, during the learning process, the hyperspace is constantly contracted to approximate the global optimal solution. Third, the convergence property of HHGLA-based RBFNN is thoroughly analyzed to ensure its application success. Finally, empirical comparisons of the proposed HHGLA-RBFNN are given to illustrate its superiority. The simulation results show the feasibility of HHGLA-RBFNN of accuracy.

Shear wave velocity prediction based on 1DCNN-BiLSTM network with attention mechanism

The Shear wave (S-wave) velocity is an essential parameter in reservoir characterization and evaluation, fluid identification, and prestack inversion. However, the cost of obtaining S-wave velocities directly from dipole acoustic logging is relatively high. At the same time, conventional data-driven S-wave velocity prediction methods exhibit several limitations, such as poor accuracy and generalization of empirical formulas, inadequate exploration of logging curve patterns of traditional fully connected neural networks, and gradient explosion and gradient vanishing problems of recurrent neural networks (RNNs). In this study, we present a reliable and low-cost deep learning (DL) approach for S-wave velocity prediction from real logging data to facilitate the solution of these problems. We designed a new network sensitive to depth sequence logging data using conventional neural networks. The new network is composed of one-dimensional (1D) convolutional, bidirectional long short-term memory (BiLSTM), attention, and fully connected layers. First, the network extracts the local features of the logging curves using a 1D convolutional layer, and then extracts the long-term sequence features of the logging curves using the BiLSTM layer, while adding an attention layer behind the BiLSTM network to further highlight the features that are more significant for S-wave velocity prediction and minimize the influence of other features to improve the accuracy of S-wave velocity prediction. Afterward, the nonlinear mapping relationship between logging data and S-wave velocity is established using several fully connected layers. We applied the new network to real field data and compared its performance with three traditional methods, including a long short-term memory (LSTM) network, a back-propagation neural network (BPNN), and an empirical formula. The performance of the four methods was quantified in terms of their coefficient of determination (R2), root mean square error (RMSE), and mean absolute error (MAE). The new network exhibited better performance and generalization ability, with R2 greater than 0.95 (0.9546, 0.9752, and 0.9680, respectively), RMSE less than 57 m/s (56.29, 23.18, and 30.17 m/s, respectively), and MAE less than 35 m/s (34.68, 16.49, and 21.47 m/s, respectively) for the three wells. The test results demonstrate the efficacy of the proposed approach, which has the potential to be widely applied in real areas where S-wave velocity logging data are not available. Furthermore, the findings of this study can help for a better understanding of the superiority of deep learning schemes and attention mechanisms for logging parameter prediction.

Hybrid Neural Networks for Solving Fully Coupled, High-Dimensional Forward–Backward Stochastic Differential Equations

The theory of forward–backward stochastic differential equations occupies an important position in stochastic analysis and practical applications. However, the numerical solution of forward–backward stochastic differential equations, especially for high-dimensional cases, has stagnated. The development of deep learning provides ideas for its high-dimensional solution. In this paper, our focus lies on the fully coupled forward–backward stochastic differential equation. We design a neural network structure tailored to the characteristics of the equation and develop a hybrid BiGRU model for solving it. We introduce the time dimension based on the sequence nature after discretizing the FBSDE. By considering the interactions between preceding and succeeding time steps, we construct the BiGRU hybrid model. This enables us to effectively capture both long- and short-term dependencies, thus mitigating issues such as gradient vanishing and explosion. Residual learning is introduced within the neural network at each time step; the structure of the loss function is adjusted according to the properties of the equation. The model established above can effectively solve fully coupled forward–backward stochastic differential equations, effectively avoiding the effects of dimensional catastrophe, gradient vanishing, and gradient explosion problems, with higher accuracy, stronger stability, and stronger model interpretability.

Analysis of Electroosmotically Modulated Peristaltic Transport of Third Grade Fluid in a Microtube Considering Slip-Dependent Zeta Potential

Abstract The present study investigates electro-osmotically modulated peristaltic transport of third-grade fluid through a microtube taking into consideration the intricate coupling of zeta potential and hydrodynamic slippage. The analytical results encompass the mathematical expressions for dimensionless electrical potential distribution as well as series solutions for stream function and axial pressure gradient up to first order utilizing the perturbation technique for small Deborah number coupled with the Cauchy product for infinite series. Critical values and ranges of wavelength have been obtained where the axial pressure gradient vanishes. Moreover, pivotal values and ranges of wavelength have also been noted for the invariance of pressure gradient with respect to Deborah number as well as Debye–Hückel parameter. Trapping phenomenon has also been investigated by contours of streamlines wherein the zones of recirculation or trapped boluses are formed predominantly near the microtube walls. Additionally, the relative enhancement in hydrodynamic slippage amplifies the trapped bolus size, whereas a diminishing behavior on bolus size is observed by the electro-osmotic parameter.

Toward Quantized Model Parallelism for Graph-Augmented MLPs Based on Gradient-Free ADMM Framework.

While graph neural networks (GNNs) are popular in the deep learning community, they suffer from several challenges including over-smoothing, over-squashing, and gradient vanishing. Recently, a series of models have attempted to relieve these issues by first augmenting the node features and then imposing node-wise functions based on multilayer perceptron (MLP), which are widely referred to as graph-augmented MLP (GA-MLP) models. However, while GA-MLP models enjoy deeper architectures for better accuracy, their efficiency largely deteriorates. Moreover, popular acceleration techniques such as stochastic-version or data-parallelism cannot be effectively applied due to the dependency among samples (i.e., nodes) in graphs. To address these issues, in this article, instead of data parallelism, we propose a parallel graph deep learning Alternating Direction Method of Multipliers (pdADMM-G) framework to achieve model parallelism: parameters in each layer of GA-MLP models can be updated in parallel. The extended pdADMM-G-Q algorithm reduces communication costs by introducing the quantization technique. Theoretical convergence to a (quantized) stationary point of the pdADMM-G algorithm and the pdADMM-G-Q algorithm is provided with a sublinear convergence rate o(1/k) , where k is the number of iterations. Extensive experiments demonstrate the convergence of two proposed algorithms. Moreover, they lead to a more massive speedup and better performance than all state-of-the-art comparison methods on nine benchmark datasets. Last but not least, the proposed pdADMM-G-Q algorithm reduces communication overheads by up to 45% without loss of performance. Our code is available at https://github.com/xianggebenben/pdADMM-G.

Enhanced physics‐informed neural networks for efficient modelling of hydrodynamics in river networks

AbstractThis article enhances the physics‐informed neural networks (PINNs) method to effectively model the hydrodynamics of real‐world river networks with irregular cross‐sections. First, we pre‐process hydraulic parameters to optimize training speed without compromising accuracy, achieving a 91.67% acceleration compared with traditional methods. To address the vanishing gradient problem, layer normalization is also incorporated into the architecture. We also introduce novel physical constraints—water level range and junction node equations—to ensure effective training convergence and enrich the model with additional physical insights. Two practical case studies using HEC‐RAS benchmarks demonstrate that our improved PINN method can predict river network hydrodynamics with less data and is less sensitive to time step size, allowing for longer computational time steps. Incorporating physical knowledge, our enhanced PINN methodology emerges as an efficient and promising avenue for modelling the complexities of hydrodynamic processes in natural river networks.

Predicting Vehicle Pose in Six Degrees of Freedom from Single Image in Real-World Traffic Environments Using Deep Pretrained Convolutional Networks and Modified Centernet

Abstract The study focuses on intelligent driving, emphasizing the importance of recognizing nearby vehicles and estimating their positions using visual input from a single image. It employs transfer learning techniques, integrating deep convolutional networks’ features into a modified CenterNet model for six-degrees-of-freedom (6DoF) vehicle position estimation. To address the vanishing gradient problem, the model incorporates simultaneous double convolutional blocks with skip connections. Utilizing the ApolloCar3D dataset, which surpasses KITTI in comprehensiveness, the study evaluates pretrained models’ performance using mean average precision (mAP). The recommended model, Center-DenseNet201, achieves a mAP of 11.82% for relative translation thresholds (A3DP-Rel) and 39.92% for absolute translation thresholds (A3DP-Abs). These findings highlight the effectiveness of pretrained models in the modified architecture, enhancing vehicle posture prediction accuracy from single images. The research contributes to autonomous vehicle development, fostering safer and more efficient navigation systems in real-world traffic scenarios.

Pneumonia Detection using CNN, Resnet and DenseNet

Abstract: Pneumonia is a prevalent respiratory infection that poses a significant threat to public health. Timely and precise diagnosis is essential for effective treatment and patient management. The proposed methodology involves training a CNN (Convolutional Neural Networks), ResNet-50 and DenseNet-121, on a large dataset of chest X ray images. CNNs play a crucial role in feature extraction from chest X-ray images for various computer vision tasks, including image recognition. The CNN model automatically learns and extracts essential visual features from the input images, capturing patterns and characteristics. On the other hand, ResNet-50 and DenseNet-121 leverage their effectiveness in handling deeper architectures and handling vanishing gradient problems. We compare our approach with existing methods to assess the quality and accuracy of the generated captions. The models undergo training and testing utilizing an extensive dataset comprising chest X-ray images, demonstrating high accuracy in detecting pneumonia, potentially offering a valuable tool for early diagnosis and treatment. The proposed pneumonia detection framework holds great promise for assisting healthcare professionals in diagnosing and treating pneumonia, thereby improving patient outcomes, and reducing healthcare costs.

Stock Market Price Prediction and Forecasting Using Stacked LSTM

Stock price movement is non-linear and complex. Several research works have been carried out to predict stock prices. Traditional approaches such as Linear Regression and Support Vector Regression were used but accuracy was not adequate. Researchers have tried to improve stock price prediction using ARIMA. Due to very high variations in stock prices, deep learning techniques are applied due to its proven accuracy in various analytics fields. Artificial Neural Network was deployed to predict stock prices but as stock prices are time-series based, recurrent neural network was applied to further improve prediction accuracy. Therefore, data scientists and analysts found that Deep Learning outperformed Machine Learning which is also proofed by all the collected research papers, and it is the most suitable methodologies to apply to the stock market forecasting domain. This paper explores different stacked LSTM Models for non-stationary financial time series in stock price prediction. This study is to predict stock market prices to make more acquaint and precise investment decisions. The experimental result will show that, this method can get quite accurate result, particularly effective in stock prediction. The proposed LSTM model will be designed to overcome gradient explosion, gradient vanishing, and save long-term memory. Firstly, The Model will have a comparative analysis between different LSTM models and integrating the model based on the result obtained from the analysis. The results will suggest that the developed stacked LSTM produces better predictive power and generalization. Key Words: stock market prediction, stacked LSTM, Deep Learning, Data pre-processing techniques, Data normalization, Neural Networks, Data Visualization, Training and Testing Set, Time series analysis, Predictive models, Future Forecasting.

Short-term wind power forecasting through stacked and bi directional LSTM techniques.

Computational intelligence (CI) based prediction models increase the efficient and effective utilization of resources for wind prediction. However, the traditional recurrent neural networks (RNN) are difficult to train on data having long-term temporal dependencies, thus susceptible to an inherent problem of vanishing gradient. This work proposed a method based on an advanced version of RNN known as long short-term memory (LSTM) architecture, which updates recurrent weights to overcome the vanishing gradient problem. This, in turn, improves training performance. The RNN model is developed based on stack LSTM and bidirectional LSTM. The parameters like mean absolute error (MAE), standard deviation error (SDE), and root mean squared error (RMSE) are utilized as performance measures for comparison with recent state-of-the-art techniques. Results showed that the proposed technique outperformed the existing techniques in terms of RMSE and MAE against all the used wind farm datasets. Whereas, a reduction in SDE is observed for larger wind farm datasets. The proposed RNN approach performed better than the existing models despite fewer parameters. In addition, the approach requires minimum processing power to achieve compatible results.

Enhancing traffic flow prediction: Dual-branch graph convolutional network with rough data inference and adaptive spatial dependencies

Traffic flow prediction is a significant application of deep learning in spatio-temporal forecasting analysis. Existing research faces challenges that hinder prediction accuracy. One challenge is the inadequate capturing of spatial dependencies in traffic flow due to fixed pre-defined graph structures. Moreover, manually designed prior graphs still have limitations in extracting spatial features. Another challenge is the instability in short-term prediction performance when pre-defined graphs are completely abandoned in favor of parameter training. Additionally, ordinary RNN sequence convolution methods also struggle to capture long-term sequential patterns in large historical traffic data, leading to gradient vanishing or exploding issues. To address these challenges, we proposes a graph convolutional network for traffic flow prediction. We combine an improved prior graph with an adaptive graph to form a dual-branch spatio-temporal neural network. In the first branch, we introduce a time graph based on rough data inference to complement the predefined static graph. In the second branch, we construct an adaptive learning framework that dynamically learns the adjacency matrix and captures global road information. By utilizing enhanced multi-scale gated convolution, we extract spatio-temporal dependencies. Our method surpasses most baseline approaches according to extensive experiments conducted on public datasets.

A lightweight and gradient-stable neural layer

To enhance resource efficiency and model deployability of neural networks, we propose a neural-layer architecture based on Householder weighting and absolute-value activating, called Householder-absolute neural layer or simply Han-layer. Compared to a fully connected layer with d-neurons and d outputs, a Han-layer reduces the number of parameters and the corresponding computational complexity from O(d2) to O(d). The Han-layer structure guarantees that the Jacobian of the layer function is always orthogonal, thus ensuring gradient stability (i.e., free of gradient vanishing or exploding issues) for any Han-layer sub-networks. Extensive numerical experiments show that one can strategically use Han-layers to replace fully connected (FC) layers, reducing the number of model parameters while maintaining or even improving the generalization performance. We will also showcase the capabilities of the Han-layer architecture on a few small stylized models, and discuss its current limitations.

Neural Oscillators for Generalization of Physics-Informed Machine Learning

A primary challenge of physics-informed machine learning (PIML) is its generalization beyond the training domain, especially when dealing with complex physical problems represented by partial differential equations (PDEs). This paper aims to enhance the generalization capabilities of PIML, facilitating practical, real-world applications where accurate predictions in unexplored regions are crucial. We leverage the inherent causality and temporal sequential characteristics of PDE solutions to fuse PIML models with recurrent neural architectures based on systems of ordinary differential equations, referred to as neural oscillators. Through effectively capturing long-time dependencies and mitigating the exploding and vanishing gradient problem, neural oscillators foster improved generalization in PIML tasks. Extensive experimentation involving time-dependent nonlinear PDEs and biharmonic beam equations demonstrates the efficacy of the proposed approach. Incorporating neural oscillators outperforms existing state-of-the-art methods on benchmark problems across various metrics. Consequently, the proposed method improves the generalization capabilities of PIML, providing accurate solutions for extrapolation and prediction beyond the training data.

Bi-ViT: Pushing the Limit of Vision Transformer Quantization

Vision transformers (ViTs) quantization offers a promising prospect to facilitate deploying large pre-trained networks on resource-limited devices. Fully-binarized ViTs (Bi-ViT) that pushes the quantization of ViTs to its limit remain largely unexplored and a very challenging task yet, due to their unacceptable performance. Through extensive empirical analyses, we identify the severe drop in ViT binarization is caused by attention distortion in self-attention, which technically stems from the gradient vanishing and ranking disorder. To address these issues, we first introduce a learnable scaling factor to reactivate the vanished gradients and illustrate its effectiveness through theoretical and experimental analyses. We then propose a ranking-aware distillation method to rectify the disordered ranking in a teacher-student framework. Bi-ViT achieves significant improvements over popular DeiT and Swin backbones in terms of Top-1 accuracy and FLOPs. For example, with DeiT-Tiny and Swin-Tiny, our method significantly outperforms baselines by 22.1% and 21.4% respectively, while 61.5x and 56.1x theoretical acceleration in terms of FLOPs compared with real-valued counterparts on ImageNet. Our codes and models are attached on https://github.com/YanjingLi0202/Bi-ViT/ .

EigenGAN: An SVD subspace-based learning for image generation using Conditional GAN

Generative adversarial networks (GANs) represent a significant advance in the field of deep learning for image generation problems. With their ability to generate highly realistic and diverse images, GANs are quickly becoming the preferred technique for a wide range of applications. However, GANs come with several limitations and challenges, chief among which are their collapse mode and instability during training. In this paper, we propose a novel generator GAN model that incorporates the singular value decomposition (SVD) process into the decoder module. The SVD integration allows the generator to recognize the underlying structures in the feature space, resulting in more robust and effective image generation. This is achieved by adjusting the singular values during the training process, effectively optimizing the SVD-based generator to produce images that match the ground truth. Another advantage of SVD integration is its ability to perform discriminative spatial decomposition that clearly reflects the differences between the generated features and the target features. By including SVD in the generator model, the resulting loss values are higher and less prone to gradient vanishing than conventional GANs, such as Pix2Pix. Our proposed SVD-based generator model can be integrated into any auto-encoder architecture, making it as a generic and versatile solution for various image-generation tasks. The effectiveness of the proposed SVD-based GAN in image generation has been validated in three challenging benchmarks for image restoration and visible-to-infrared image translation. Extensive experiments demonstrate the significant quantitative and qualitative improvements achieved by our SVD-based GAN compared to baseline GAN architectures. The overall system outperforms the current state of the art in all benchmarks tested.

DLGAN: Undersampled MRI reconstruction using Deep Learning based Generative Adversarial Network

Magnetic Resonance Imaging (MRI) is a crucial tool for quantitative image analysis and clinical diagnosis, providing detailed anatomical images to assist in the detection of various abnormalities. However, the widespread use of MRI is hindered by the challenges associated with long sampling periods and low-resolution image processing, prompting the development of various traditional methods to address these limitations. In recent times, advanced Deep Learning (DL) techniques have been applied to tackle the inverse problem of reconstructing MRI images from undersampled data. These DL models have demonstrated substantial improvements in terms of image reconstruction performance, cost-effectiveness, and reduced acquisition time, offering significant potential for further enhancements in this domain. This study introduces a novel DLGAN (Deep Learning Generative Adversarial Network) model comprising two sub-GAN modules tailored for specific datasets. Each module incorporates DLGAN (Generator Block, GB) and DLGAN (Discriminator Block, DB) blocks, strategically designed to regenerate MR images from K Space data. These blocks effectively leverage information from both the ground truth and KSpace characteristics, resulting in enhanced image reconstruction performance while also reducing complexity and improving overall efficiency. The DLGAN model effectively addresses common issues such as artefact removal and vanishing-gradient problems through the extraction of hierarchical features. To evaluate the model’s effectiveness, comprehensive experiments were conducted, utilizing metrics such as Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM). The experimental findings demonstrate that the proposed DLGAN model outperforms the most recent designs in MRI image reconstruction, showcasing its potential for significant advancements in clinical diagnosis and quantitative image analysis.

A novel hybrid STL-transformer-ARIMA architecture for aviation failure events prediction

Accurate prediction of aviation failure events helps to anticipate future safety situations and protect against further uncontrollable accidents. However, the large sample size, complex temporal characteristics, and significant long-term correlation of aviation failure events increase the operational cost of accurate prediction. To address these challenges, this paper proposes a novel approach involving seasonal-trend decomposition using Loess (STL) and a hybrid prediction model consisting of a transformer and autoregressive integrated moving average (ARIMA). First, STL decomposition is utilized to isolate trend, seasonal, and remainder components, contributing to a comprehensive understanding of the events sample characteristics. The trend component is then trained and predicted using transformer, solving the vanishing gradient problem and improving computational efficiency. ARIMA is employed to train and predict the seasonal and remainder components, maintaining accuracy while reducing complexity. Finally, a comparative evaluation between the proposed and multiple existing approaches is conducted using Aviation Safety Reporting System (ASRS) data. The results demonstrate that the STL-transformer-ARIMA provides more accurate predictions of failure events than single model. It also exhibits significant advantages in robustness and generalization capacity compared to single transformer-based predictors. This revealed that the proposed approach performed better in predicting aviation failure events.