Vanishing Gradient Research Articles

Predicting Vehicle Pose in Six Degrees of Freedom from Single Image in Real-World Traffic Environments Using Deep Pretrained Convolutional Networks and Modified Centernet

Abstract The study focuses on intelligent driving, emphasizing the importance of recognizing nearby vehicles and estimating their positions using visual input from a single image. It employs transfer learning techniques, integrating deep convolutional networks’ features into a modified CenterNet model for six-degrees-of-freedom (6DoF) vehicle position estimation. To address the vanishing gradient problem, the model incorporates simultaneous double convolutional blocks with skip connections. Utilizing the ApolloCar3D dataset, which surpasses KITTI in comprehensiveness, the study evaluates pretrained models’ performance using mean average precision (mAP). The recommended model, Center-DenseNet201, achieves a mAP of 11.82% for relative translation thresholds (A3DP-Rel) and 39.92% for absolute translation thresholds (A3DP-Abs). These findings highlight the effectiveness of pretrained models in the modified architecture, enhancing vehicle posture prediction accuracy from single images. The research contributes to autonomous vehicle development, fostering safer and more efficient navigation systems in real-world traffic scenarios.

Pneumonia Detection using CNN, Resnet and DenseNet

Abstract: Pneumonia is a prevalent respiratory infection that poses a significant threat to public health. Timely and precise diagnosis is essential for effective treatment and patient management. The proposed methodology involves training a CNN (Convolutional Neural Networks), ResNet-50 and DenseNet-121, on a large dataset of chest X ray images. CNNs play a crucial role in feature extraction from chest X-ray images for various computer vision tasks, including image recognition. The CNN model automatically learns and extracts essential visual features from the input images, capturing patterns and characteristics. On the other hand, ResNet-50 and DenseNet-121 leverage their effectiveness in handling deeper architectures and handling vanishing gradient problems. We compare our approach with existing methods to assess the quality and accuracy of the generated captions. The models undergo training and testing utilizing an extensive dataset comprising chest X-ray images, demonstrating high accuracy in detecting pneumonia, potentially offering a valuable tool for early diagnosis and treatment. The proposed pneumonia detection framework holds great promise for assisting healthcare professionals in diagnosing and treating pneumonia, thereby improving patient outcomes, and reducing healthcare costs.

Stock Market Price Prediction and Forecasting Using Stacked LSTM

Stock price movement is non-linear and complex. Several research works have been carried out to predict stock prices. Traditional approaches such as Linear Regression and Support Vector Regression were used but accuracy was not adequate. Researchers have tried to improve stock price prediction using ARIMA. Due to very high variations in stock prices, deep learning techniques are applied due to its proven accuracy in various analytics fields. Artificial Neural Network was deployed to predict stock prices but as stock prices are time-series based, recurrent neural network was applied to further improve prediction accuracy. Therefore, data scientists and analysts found that Deep Learning outperformed Machine Learning which is also proofed by all the collected research papers, and it is the most suitable methodologies to apply to the stock market forecasting domain. This paper explores different stacked LSTM Models for non-stationary financial time series in stock price prediction. This study is to predict stock market prices to make more acquaint and precise investment decisions. The experimental result will show that, this method can get quite accurate result, particularly effective in stock prediction. The proposed LSTM model will be designed to overcome gradient explosion, gradient vanishing, and save long-term memory. Firstly, The Model will have a comparative analysis between different LSTM models and integrating the model based on the result obtained from the analysis. The results will suggest that the developed stacked LSTM produces better predictive power and generalization. Key Words: stock market prediction, stacked LSTM, Deep Learning, Data pre-processing techniques, Data normalization, Neural Networks, Data Visualization, Training and Testing Set, Time series analysis, Predictive models, Future Forecasting.

Short-term wind power forecasting through stacked and bi directional LSTM techniques.

Computational intelligence (CI) based prediction models increase the efficient and effective utilization of resources for wind prediction. However, the traditional recurrent neural networks (RNN) are difficult to train on data having long-term temporal dependencies, thus susceptible to an inherent problem of vanishing gradient. This work proposed a method based on an advanced version of RNN known as long short-term memory (LSTM) architecture, which updates recurrent weights to overcome the vanishing gradient problem. This, in turn, improves training performance. The RNN model is developed based on stack LSTM and bidirectional LSTM. The parameters like mean absolute error (MAE), standard deviation error (SDE), and root mean squared error (RMSE) are utilized as performance measures for comparison with recent state-of-the-art techniques. Results showed that the proposed technique outperformed the existing techniques in terms of RMSE and MAE against all the used wind farm datasets. Whereas, a reduction in SDE is observed for larger wind farm datasets. The proposed RNN approach performed better than the existing models despite fewer parameters. In addition, the approach requires minimum processing power to achieve compatible results.

Enhancing traffic flow prediction: Dual-branch graph convolutional network with rough data inference and adaptive spatial dependencies

Traffic flow prediction is a significant application of deep learning in spatio-temporal forecasting analysis. Existing research faces challenges that hinder prediction accuracy. One challenge is the inadequate capturing of spatial dependencies in traffic flow due to fixed pre-defined graph structures. Moreover, manually designed prior graphs still have limitations in extracting spatial features. Another challenge is the instability in short-term prediction performance when pre-defined graphs are completely abandoned in favor of parameter training. Additionally, ordinary RNN sequence convolution methods also struggle to capture long-term sequential patterns in large historical traffic data, leading to gradient vanishing or exploding issues. To address these challenges, we proposes a graph convolutional network for traffic flow prediction. We combine an improved prior graph with an adaptive graph to form a dual-branch spatio-temporal neural network. In the first branch, we introduce a time graph based on rough data inference to complement the predefined static graph. In the second branch, we construct an adaptive learning framework that dynamically learns the adjacency matrix and captures global road information. By utilizing enhanced multi-scale gated convolution, we extract spatio-temporal dependencies. Our method surpasses most baseline approaches according to extensive experiments conducted on public datasets.

A lightweight and gradient-stable neural layer

To enhance resource efficiency and model deployability of neural networks, we propose a neural-layer architecture based on Householder weighting and absolute-value activating, called Householder-absolute neural layer or simply Han-layer. Compared to a fully connected layer with d-neurons and d outputs, a Han-layer reduces the number of parameters and the corresponding computational complexity from O(d2) to O(d). The Han-layer structure guarantees that the Jacobian of the layer function is always orthogonal, thus ensuring gradient stability (i.e., free of gradient vanishing or exploding issues) for any Han-layer sub-networks. Extensive numerical experiments show that one can strategically use Han-layers to replace fully connected (FC) layers, reducing the number of model parameters while maintaining or even improving the generalization performance. We will also showcase the capabilities of the Han-layer architecture on a few small stylized models, and discuss its current limitations.

Neural Oscillators for Generalization of Physics-Informed Machine Learning

A primary challenge of physics-informed machine learning (PIML) is its generalization beyond the training domain, especially when dealing with complex physical problems represented by partial differential equations (PDEs). This paper aims to enhance the generalization capabilities of PIML, facilitating practical, real-world applications where accurate predictions in unexplored regions are crucial. We leverage the inherent causality and temporal sequential characteristics of PDE solutions to fuse PIML models with recurrent neural architectures based on systems of ordinary differential equations, referred to as neural oscillators. Through effectively capturing long-time dependencies and mitigating the exploding and vanishing gradient problem, neural oscillators foster improved generalization in PIML tasks. Extensive experimentation involving time-dependent nonlinear PDEs and biharmonic beam equations demonstrates the efficacy of the proposed approach. Incorporating neural oscillators outperforms existing state-of-the-art methods on benchmark problems across various metrics. Consequently, the proposed method improves the generalization capabilities of PIML, providing accurate solutions for extrapolation and prediction beyond the training data.

Bi-ViT: Pushing the Limit of Vision Transformer Quantization

Vision transformers (ViTs) quantization offers a promising prospect to facilitate deploying large pre-trained networks on resource-limited devices. Fully-binarized ViTs (Bi-ViT) that pushes the quantization of ViTs to its limit remain largely unexplored and a very challenging task yet, due to their unacceptable performance. Through extensive empirical analyses, we identify the severe drop in ViT binarization is caused by attention distortion in self-attention, which technically stems from the gradient vanishing and ranking disorder. To address these issues, we first introduce a learnable scaling factor to reactivate the vanished gradients and illustrate its effectiveness through theoretical and experimental analyses. We then propose a ranking-aware distillation method to rectify the disordered ranking in a teacher-student framework. Bi-ViT achieves significant improvements over popular DeiT and Swin backbones in terms of Top-1 accuracy and FLOPs. For example, with DeiT-Tiny and Swin-Tiny, our method significantly outperforms baselines by 22.1% and 21.4% respectively, while 61.5x and 56.1x theoretical acceleration in terms of FLOPs compared with real-valued counterparts on ImageNet. Our codes and models are attached on https://github.com/YanjingLi0202/Bi-ViT/ .

EigenGAN: An SVD subspace-based learning for image generation using Conditional GAN

Generative adversarial networks (GANs) represent a significant advance in the field of deep learning for image generation problems. With their ability to generate highly realistic and diverse images, GANs are quickly becoming the preferred technique for a wide range of applications. However, GANs come with several limitations and challenges, chief among which are their collapse mode and instability during training. In this paper, we propose a novel generator GAN model that incorporates the singular value decomposition (SVD) process into the decoder module. The SVD integration allows the generator to recognize the underlying structures in the feature space, resulting in more robust and effective image generation. This is achieved by adjusting the singular values during the training process, effectively optimizing the SVD-based generator to produce images that match the ground truth. Another advantage of SVD integration is its ability to perform discriminative spatial decomposition that clearly reflects the differences between the generated features and the target features. By including SVD in the generator model, the resulting loss values are higher and less prone to gradient vanishing than conventional GANs, such as Pix2Pix. Our proposed SVD-based generator model can be integrated into any auto-encoder architecture, making it as a generic and versatile solution for various image-generation tasks. The effectiveness of the proposed SVD-based GAN in image generation has been validated in three challenging benchmarks for image restoration and visible-to-infrared image translation. Extensive experiments demonstrate the significant quantitative and qualitative improvements achieved by our SVD-based GAN compared to baseline GAN architectures. The overall system outperforms the current state of the art in all benchmarks tested.

DLGAN: Undersampled MRI reconstruction using Deep Learning based Generative Adversarial Network

Magnetic Resonance Imaging (MRI) is a crucial tool for quantitative image analysis and clinical diagnosis, providing detailed anatomical images to assist in the detection of various abnormalities. However, the widespread use of MRI is hindered by the challenges associated with long sampling periods and low-resolution image processing, prompting the development of various traditional methods to address these limitations. In recent times, advanced Deep Learning (DL) techniques have been applied to tackle the inverse problem of reconstructing MRI images from undersampled data. These DL models have demonstrated substantial improvements in terms of image reconstruction performance, cost-effectiveness, and reduced acquisition time, offering significant potential for further enhancements in this domain. This study introduces a novel DLGAN (Deep Learning Generative Adversarial Network) model comprising two sub-GAN modules tailored for specific datasets. Each module incorporates DLGAN (Generator Block, GB) and DLGAN (Discriminator Block, DB) blocks, strategically designed to regenerate MR images from K Space data. These blocks effectively leverage information from both the ground truth and KSpace characteristics, resulting in enhanced image reconstruction performance while also reducing complexity and improving overall efficiency. The DLGAN model effectively addresses common issues such as artefact removal and vanishing-gradient problems through the extraction of hierarchical features. To evaluate the model’s effectiveness, comprehensive experiments were conducted, utilizing metrics such as Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM). The experimental findings demonstrate that the proposed DLGAN model outperforms the most recent designs in MRI image reconstruction, showcasing its potential for significant advancements in clinical diagnosis and quantitative image analysis.

A novel hybrid STL-transformer-ARIMA architecture for aviation failure events prediction

Accurate prediction of aviation failure events helps to anticipate future safety situations and protect against further uncontrollable accidents. However, the large sample size, complex temporal characteristics, and significant long-term correlation of aviation failure events increase the operational cost of accurate prediction. To address these challenges, this paper proposes a novel approach involving seasonal-trend decomposition using Loess (STL) and a hybrid prediction model consisting of a transformer and autoregressive integrated moving average (ARIMA). First, STL decomposition is utilized to isolate trend, seasonal, and remainder components, contributing to a comprehensive understanding of the events sample characteristics. The trend component is then trained and predicted using transformer, solving the vanishing gradient problem and improving computational efficiency. ARIMA is employed to train and predict the seasonal and remainder components, maintaining accuracy while reducing complexity. Finally, a comparative evaluation between the proposed and multiple existing approaches is conducted using Aviation Safety Reporting System (ASRS) data. The results demonstrate that the STL-transformer-ARIMA provides more accurate predictions of failure events than single model. It also exhibits significant advantages in robustness and generalization capacity compared to single transformer-based predictors. This revealed that the proposed approach performed better in predicting aviation failure events.

Improved Generative Adversarial Network for Bearing Fault Diagnosis with a Small Number of Data and Unbalanced Data

Fault data under real operating conditions are often difficult to collect, making the number of trained fault data small and out of proportion to normal data. Thus, fault diagnosis symmetry (balance) is compromised. This will result in less effective fault diagnosis methods for cases with a small number of data and data imbalances (S&I). We present an innovative solution to overcome this problem, which is composed of two components: data augmentation and fault diagnosis. In the data augmentation section, the S&I dataset is supplemented with a deep convolutional generative adversarial network based on a gradient penalty and Wasserstein distance (WDCGAN-GP), which solve the problems of the generative adversarial network (GAN) being prone to model collapse and the gradient vanishing during the training time. The addition of self-attention allows for a better identification and generation of sample features. Finally, the addition of spectral normalization can stabilize the training of the model. In the fault diagnosis section, fault diagnosis is performed through a convolutional neural network with coordinate attention (CNN-CA). Our experiments conducted on two bearing fault datasets for comparison demonstrate that the proposed method surpasses other comparative approaches in terms of the quality of data augmentation and the accuracy of fault diagnosis. It effectively addresses S&I fault diagnosis challenges.

AN EEG-BASED EMOTION RECOGNITION MODEL USING AN INTERACTION DESIGN FRAMEWORK AND DEEP LEARNING

This research makes significant advances in the field of emotion recognition by presenting a new generative adversarial network (GAN) model that integrates deep learning with electroencephalography (EEG). To achieve more accurate data production and real data matching, the model utilizes self-attention and residual neural networks. Additionally, this process is accomplished by substituting an autoencoder for the discriminator in the GAN, and incorporating a reconstruction loss function. We include the self-attention mechanism and residual block in the building of the model to overcome the vanishing gradient problem. This allows the model to acquire information related to emotions in a more in-depth manner, which ultimately results in an improvement in the emotion detection accuracy. The DEAP and MAHNOB-HCI datasets are chosen for the experimental validation portion of this research. These datasets are subsequently compared and analyzed with traditional deep learning methods and well-known emotion identification algorithms. Based on these findings, it is evident that the model that we propose performs exceptionally well on the emotion recognition test, which offers substantial support for studies and applications in this field. In addition, within the context of emotion detection systems, this study places particular emphasis on the crucial role that interaction design frameworks play in enhancing both the user experience and the usability of the system. By pushing the emotion recognition technology boundaries, a new paradigm for the application of deep learning in EEG emotion recognition is provided with this comprehensive research contribution.

An Improved Fault Localization Method for Direct Current Filters in HVDC Systems: Development and Application of the DRNCNN Model

This paper focus on direct current (DC) filter grounding faults to propose a novel dilated normalized residual convolutional neural network (DRNCNN) fault diagnosis model for high-voltage direct current (HVDC) transmission systems. To address the insufficiency of the traditional model’s receptive field in dealing with high-dimensional and nonlinear data, this paper incorporates dilated convolution and batch normalization (BN), significantly enhancing the CNN’s capability to capture complex spatial features. Furthermore, this paper integrates residual connections and parameter rectified linear units (PReLU) to optimize gradient propagation and mitigate the issue of gradient vanishing during training. These innovative improvements, embodied in the DRNCNN model, substantially increase the accuracy of fault detection, achieving a diagnostic accuracy rate of 99.28%.

Seeing the world from its words: All-embracing Transformers for fingerprint-based indoor localization

In this paper, we present all-embracing Transformers (AaTs) that are capable of deftly manipulating attention mechanism for Received Signal Strength (RSS) fingerprints in order to invigorate localizing performance. Since most machine learning models applied to the RSS modality do not possess any attention mechanism, they can merely capture superficial representations. Moreover, compared to textual and visual modalities, the RSS modality is inherently notorious for its sensitivity to environmental dynamics. Such adversities inhibit their access to subtle but distinct representations that characterize the corresponding location, ultimately resulting in significant degradation in the testing phase. In contrast, a major appeal of AaTs is the ability to focus exclusively on relevant anchors in RSS sequences, allowing full rein to the exploitation of subtle and distinct representations for specific locations. This also facilitates disregarding redundant clues formed by noisy ambient conditions, thus enhancing accuracy in localization. Apart from that, explicitly resolving the representation collapse (i.e., none-informative or homogeneous features, and gradient vanishing) can further invigorate the self-attention process in transformer blocks, by which subtle but distinct representations to specific locations are radically captured with ease. For that purpose, we first enhance our proposed model with two sub-constraints, namely covariance and variance losses at the Anchor2Vec. The proposed constraints are automatically mediated with the primary task towards a novel multi-task learning manner. In an advanced manner, we present further the ultimate in design with a few simple tweaks carefully crafted for transformer encoder blocks. This effort aims to promote representation augmentation via stabilizing the inflow of gradients to these blocks. Thus, the problems of representation collapse in regular Transformers can be tackled. To evaluate our AaTs, we compare the models with the state-of-the-art (SoTA) methods on three benchmark indoor localization datasets. The experimental results confirm our hypothesis and show that our proposed models could deliver much higher and more stable accuracy.

RAAWC-UNet: an apple leaf and disease segmentation method based on residual attention and atrous spatial pyramid pooling improved UNet with weight compression loss.

Early detection of leaf diseases is necessary to control the spread of plant diseases, and one of the important steps is the segmentation of leaf and disease images. The uneven light and leaf overlap in complex situations make segmentation of leaves and diseases quite difficult. Moreover, the significant differences in ratios of leaf and disease pixels results in a challenge in identifying diseases. To solve the above issues, the residual attention mechanism combined with atrous spatial pyramid pooling and weight compression loss of UNet is proposed, which is named RAAWC-UNet. Firstly, weights compression loss is a method that introduces a modulation factor in front of the cross-entropy loss, aiming at solving the problem of the imbalance between foreground and background pixels. Secondly, the residual network and the convolutional block attention module are combined to form Res_CBAM. It can accurately localize pixels at the edge of the disease and alleviate the vanishing of gradient and semantic information from downsampling. Finally, in the last layer of downsampling, the atrous spatial pyramid pooling is used instead of two convolutions to solve the problem of insufficient spatial context information. The experimental results show that the proposed RAAWC-UNet increases the intersection over union in leaf and disease segmentation by 1.91% and 5.61%, and the pixel accuracy of disease by 4.65% compared with UNet. The effectiveness of the proposed method was further verified by the better results in comparison with deep learning methods with similar network architectures.

A two-branch multiscale spectral-spatial feature extraction network for hyperspectral image classification

In the field of hyperspectral image (HSI) classification in remote sensing, the combination of spectral and spatial features has gained considerable attention. In addition, the multiscale feature extraction approach is very effective at improving the classification accuracy for HSIs, capable of capturing a large amount of intrinsic information. However, some existing methods for extracting spectral and spatial features can only generate low-level features and consider limited scales, leading to low classification results, and dense-connection based methods enhance the feature propagation at the cost of high model complexity. This paper presents a two-branch multiscale spectral-spatial feature extraction network (TBMSSN) for HSI classification. We design the multiscale spectral feature extraction (MSEFE) and multiscale spatial feature extraction (MSAFE) modules to improve the feature representation, and a spatial attention mechanism is applied in the MSAFE module to reduce redundant information and enhance the representation of spatial features at multiscale. Then we densely connect series of MSEFE or MSAFE modules respectively in a two-branch framework to balance efficiency and effectiveness, alleviate the vanishing-gradient problem and strengthen the feature propagation. To evaluate the effectiveness of the proposed method, the experimental results were carried out on bench mark HSI datasets, demonstrating that TBMSSN obtained higher classification accuracy compared with several state-of-the-art methods.

Study on breast cancer image detection and classification based on residual connected convolutional neural network (CNN)

This paper presents a method for breast cancer image detection and classification based on convolutional neural networks. While traditional breast cancer detection methods carry the risk of subjectivity and misclassification, machine learning can improve the accuracy and efficiency of detection by training algorithms to automatically identify breast cancer lesions in patients. The dataset of this paper includes 7909 microscopic images of breast tumor tissues with different magnifications collected from 82 patients, which are divided into two categories: benign and malignant tumors. In this paper, residual connection is designed for convolutional neural networks, which is a cross-layer connection method that can effectively solve the gradient vanishing and gradient explosion problems in deep neural networks, and at the same time, improve the generalization ability and training speed of the model. In this paper, the dataset is divided into training set, validation set and test set according to the ratio of 6:2:2. Through training and validation, the prediction accuracy of the obtained model on the test set reaches 85.5%, which achieves good prediction results in breast cancer image detection and classification. Eventually, the model's loss converged at 0.53 and the AUC was 0.832. The research results in this paper are of great significance for the early diagnosis and treatment of breast cancer. Automatic identification and classification of breast cancer images by machine learning algorithms can reduce the subjective judgment of doctors, improve the accuracy and efficiency of detection, and provide important support for the early diagnosis and treatment of breast cancer. Meanwhile, the research method and results of this paper also provide reference and reference for image detection and classification of other malignant tumors.

Underactuated MSV path following control via stable adversarial inverse reinforcement learning

Model-based control approaches are inadequate to solve the marine surface vehicle (MSV) path-following problem, especially under adverse environments. To effectively deal with the MSV path-following problem, model-free deep reinforcement learning (DRL) based methods have been developed. However, defining an efficient reward function for DRL in path following tasks is rather difficult. Providing expert demonstration is often easier than designing effective reward functions. Thus, we propose a model-free stable adversarial inverse reinforcement learning (SAIRL) algorithm that only adopts the state of MSV and reconstructs the reward function from the expert demonstration. The SAIRL algorithm is designed to guarantee the prescribed MSV path following accuracy and training stability. It utilizes an alternative loss function and dual-discriminator framework to dissolve the issue of policy collapse, which arises due to the vanishing gradient of the discriminator. Simulations and experiments have validated that the SAIRL algorithm outperforms other baseline algorithms in terms of path-following accuracy and stability of convergence.

Face recognition technology based on ResNet-50

Face recognition technology is progressively finding its place across diverse domains. In pursuit of enhancing the efficacy of face recognition systems, this study employs a ResNet-50 deep convolutional neural network. The dataset is meticulously gathered and processed via OpenCV, thus amplifying the precision and utility of face recognition. ResNet, an advanced convolutional neural network, incorporates the concept of residual connections, bridging convolutional layers through shortcut connections. These connections facilitate the addition of input to output, forming residual blocks. Consequently, ResNet-50 efficiently tackles the vanishing gradient issue, enabling the training of exceptionally deep networks. With 49 convolutional layers and a fully connected layer, ResNet-50 boasts a robust architecture. To emulate varying brightness conditions, post-collection image adjustments are applied randomly. This strategy curbs the impact of divergent lighting scenarios on recognition accuracy, bolstering the models practical applicability. Notably, experimental outcomes underscore the commendable performance of the trained ResNet-50 model in face recognition trials. This substantiates the broad-spectrum viability of face recognition technology in domains such as security surveillance, human-machine interaction, identity verification, and beyond.