Problem Of Gradient Disappearance Research Articles

Research on advertising content recognition based on convolutional neural network and recurrent neural network

The problem to be solved in this paper is to identify the text advertisement information published by users in a medium-sized social networking website. First, the text is segmented and then the text is transformed into sequence tensor by using a word vector representation method, which is input into the deep neural network. Compared with other neural networks, RNN is good at processing training samples with continuous input sequence, and the length of the sequence is different. Although RNN can theoretically solve the training of sequential data beautifully, it has the problem of gradient disappearance, so it is a special LSTM based on RNN model that is widely used in practice. In the experiment, the convolutional neural network is used to process text sequence, and time is regarded as a spatial dimension. Finally, it briefly introduces the use of universal language model fine-tuning for text classification.

Real-time object detection based on YOLO-v2 for tiny vehicle object

Abstract In Automatic Driving System (ADS) and Driver Assistance System (DAS), object detection plays a vital part. Nevertheless, existing real-time detection models for tiny vehicle objects have the problems of low precision and poor performance. To solve these issues, we propose a novel real-time object detection model based on You Only Look Once Version 2 (YOLO-v2) deep learning framework for tiny vehicle objects, called Optimized You Only Look Once Version 2 (O-YOLO-v2). In the proposed model, a new structure is introduced to strengthen the feature extraction ability of the network by adding convolution layers at different locations. Meanwhile, the problem of gradient disappearance or dispersion caused by increasing network depth is solved by adding residual modules. Furthermore, in order to promote the detection accuracy of tiny vehicle objects, we combine the low-level features and high-level features of the network. The experimental findings and analysis on a KITTI dataset show that the model not only promotes the accuracy of tiny vehicle object detection but also improves the accuracy of vehicle detection (the accuracy reaches 94%) without decreasing the detection speed.

Retracted] Image Recognition of Badminton Swing Motion Based on Single Inertial Sensor

This article analyzes the method of reading data from inertial sensors. We introduce how to create a 3D scene and a 3D human body model and use inertial sensors to drive the 3D human body model. We capture the movement of the lower limbs of the human body when a small number of inertial sensor nodes are used. This paper introduces the idea of residual error into the deep LSTM network to solve the problem of gradient disappearance and gradient explosion. The main problem to be solved by wearable inertial sensor continuous human motion recognition is the modeling of time series. This paper chooses the LSTM network which can handle time series as well as the main frame. In order to reduce the gradient disappearance and gradient explosion problems in the deep LSTM network, the structure of the deep LSTM network is adjusted based on the residual learning idea. In this paper, a data acquisition method using a single inertial sensor fixed on the bottom of a badminton racket is proposed, and a window segmentation method based on the combination of sliding window and action window in real‐time motion data stream is proposed. We performed feature extraction on the intercepted motion data and performed dimensionality reduction. An improved Deep Residual LSTM model is designed to identify six common swing movements. The first‐level recognition algorithm uses the C4.5 decision tree algorithm to recognize the athlete’s gripping style, and the second‐level recognition algorithm uses the random forest algorithm to recognize the swing movement. Simulation experiments confirmed that the proposed improved Deep Residual LSTM algorithm has an accuracy of over 90.0% for the recognition of six common swing movements.

Multimedia Network Public Opinion Supervision Prediction Algorithm Based on Big Data

This article focuses on the multidimensional construction of the multimedia network public opinion supervision mechanism, puts the research on the background of the era of big data, and based on the analysis and definition of the difference between network public opinion and network public opinion, deeply summarizes the network public opinion in the era of big data. New features analyze the opportunities and challenges faced by online public opinion in the era of big data. Based on the rational construction of the index system, this paper studies the multimedia network public opinion evaluation and prediction algorithm. Existing network public opinion assessment and prediction algorithms have shortcomings in capturing the characteristics of data sequences and the long-term dependence of data sequences, and the problems of overfitting and gradient disappearance may occur during training. Because of the above problems, based on the long-term and short-term memory network model, a regularized method is used to construct a multimedia network public opinion prediction model algorithm. This paper builds a multimedia network public opinion threat rating evaluation model based on the public opinion supervision prediction model and conducts analysis. The model constructed this time can not only improve the accuracy of public opinion assessment and prediction but also better avoid the problem of gradient disappearance and overfitting.

Research on advertising content recognition based on convolutional neural network and recurrent neural network

The problem to be solved in this paper is to identify the text advertisement information published by users in a medium-sized social networking website. First, the text is segmented and then the text is transformed into sequence tensor by using a word vector representation method, which is input into the deep neural network. Compared with other neural networks, RNN is good at processing training samples with continuous input sequence, and the length of the sequence is different. Although RNN can theoretically solve the training of sequential data beautifully, it has the problem of gradient disappearance, so it is a special LSTM based on RNN model that is widely used in practice. In the experiment, the convolutional neural network is used to process text sequence, and time is regarded as a spatial dimension. Finally, it briefly introduces the use of universal language model fine-tuning for text classification.

Improved generative adversarial network and its application in image oil painting style transfer

In view of the difficulty in training the algorithm of image oil painting style migration and reconstruction based on the generative adversarial network, and the loss gradient of generator and discriminator disappears, this paper proposes an improved generative adversarial network based on gradient penalty, and constructs the total variance loss function to carry out the research of image oil painting style migration and reconstruction. Firstly, the Wasserstein distance (WGAN) is added to the loss function of the generative adversarial network to improve the stability of the alternative iterative training; secondly, the gradient penalty (WGAN-GP) is added to the loss function to deal with the problem of gradient disappearance in the training; finally, the LBP texture feature and total variation of the prototype are introduced based on the CycleGAN Loss noise constraint is used to improve the edge and texture strength of the image after migration of oil painting style. The experimental results show that the WGAN-GP algorithm constructed in this study has the ability of stable gradient and alternating iterative convergence, and the total variation loss noise constraint can provide good edge and texture details for the migration process of image oil painting style. Compared with the existing mainstream algorithm, the algorithm proposed in this study has better performance of image oil painting style migration and reconstruction, and better effect of image oil painting style migration and reconstruction.

Facial expression recognition based on bidirectional gated recurrent units within deep residual network

Purposerecent years, facial expression recognition has been widely used in human machine interaction, clinical medicine and safe driving. However, there is a limitation that conventional recurrent neural networks can only learn the time-series characteristics of expressions based on one-way propagation information.Design/methodology/approachTo solve such limitation, this paper proposes a novel model based on bidirectional gated recurrent unit networks (Bi-GRUs) with two-way propagations, and the theory of identity mapping residuals is adopted to effectively prevent the problem of gradient disappearance caused by the depth of the introduced network. Since the Inception-V3 network model for spatial feature extraction has too many parameters, it is prone to overfitting during training. This paper proposes a novel facial expression recognition model to add two reduction modules to reduce parameters, so as to obtain an Inception-W network with better generalization.FindingsFinally, the proposed model is pretrained to determine the best settings and selections. Then, the pretrained model is experimented on two facial expression data sets of CK+ and Oulu- CASIA, and the recognition performance and efficiency are compared with the existing methods. The highest recognition rate is 99.6%, which shows that the method has good recognition accuracy in a certain range.Originality/valueBy using the proposed model for the applications of facial expression, the high recognition accuracy and robust recognition results with lower time consumption will help to build more sophisticated applications in real world.

Depth-Wise Separable Convolution Neural Network with Residual Connection for Hyperspectral Image Classification

The neural network-based hyperspectral images (HSI) classification model has a deep structure, which leads to the increase of training parameters, long training time, and excessive computational cost. The deepened network models are likely to cause the problem of gradient disappearance, which limits further improvement for its classification accuracy. To this end, a residual unit with fewer training parameters were constructed by combining the residual connection with the depth-wise separable convolution. With the increased depth of the network, the number of output channels of each residual unit increases linearly with a small amplitude. The deepened network can continuously extract the spectral and spatial features while building a cone network structure by stacking the residual units. At the end of executing the model, a 1 × 1 convolution layer combined with a global average pooling layer can be used to replace the traditional fully connected layer to complete the classification with reduced parameters needed in the network. Experiments were conducted on three benchmark HSI datasets: Indian Pines, Pavia University, and Kennedy Space Center. The overall classification accuracy was 98.85%, 99.58%, and 99.96% respectively. Compared with other classification methods, the proposed network model guarantees a higher classification accuracy while spending less time on training and testing sample sites.

The image inpainting algorithm used on multi-scale generative adversarial networks and neighbourhood

Various problems existed in the image inpainting algorithms, which can’t meet people's requirements visually. Aiming at the defects of the existing image inpainting algorithms, such as low accuracy, poor visual consistency, and unstable training, an improved image inpainting algorithm used on a multi-scale generative adversarial network (GAN) and neighbourhood model have been proposed in the paper. The proposed algorithm mainly improves the network structure of the discriminator, and introduces a multi-scale discriminator based on the global discriminator and the local discriminator. The multi-scale discriminators were trained on images of different resolutions. Discriminators of different scales have different receptive fields, which can guide the generator to generate more global image views and finer details. Aiming at the problem of gradient disappearance or gradient explosion that often occurs in GAN training, the method of WGAN (Wasserstein GAN) has been used to simulate the sample data distribution using EM distance. The proposed model has been trained and tested on the CelebA, ImageNet, and Place2. The experimental results show that compared with the previous algorithm model, the proposed algorithm improves the accuracy of image inpainting and can generate more realistic repairing images, and it is suitable for many types of images.

Hyperspectral remote sensing image classification based on dense residual three-dimensional convolutional neural network

The invention discloses a hyperspectral remote sensing image classification method based on a dense residual three-dimensional convolutional neural network. According to the method, original hyperspectral data are used as network input, three-dimensional spatial-spectral features of a hyperspectral remote sensing image are extracted through three-dimensional convolution, the hyperspectral image can be directly processed through three-dimensional convolution, preprocessing operations such as dimension reduction are not needed, and the spatial-spectral features of the hyperspectral image are extracted more sufficiently. The dense residual network is used to deepen the number of network layers and learn deeper spectral and spatial features, the residual network can effectively reduce the problem of gradient disappearance along with the increase of the network depth, and the structure can more effectively utilize the features and enhance the feature transfer between convolutional layers. The training time is shortened through an early stop method, classification prediction is carried out through a Soft-max classifier, and an initial classification result is obtained; and proposing a multi-label conditional random field optimization algorithm, and optimizing a classification result. The method improves the operation efficiency, and improves the classification accuracy of the remotesensing images.

Deep Pyramid Convolutional Neural Network Integrated with Self-attention Mechanism and Highway Network for Text Classification

Text classification is one of the basic tasks of natural language processing. In recent years, deep learning has been widely used in text classification tasks. The representative one is the convolutional neural network. DPCNN is a deep convolutional neural network text classification model that can obtain long-distance text information, but it focuses on the extraction of global features and ignores the extraction of local features of the text. Some local feature information is very important and plays an important role in text classification tasks. Therefore, in this paper, a self-attention mechanism is introduced to extract local features of text based on the lack of extracting local features of DPCNN. In addition, although the deep convolutional neural network can extract deeper features, it is easy to cause the problem of gradient disappearance during training. Therefore, the highway network is introduced to prevent the problem of gradient disappearance caused by network training and improve the performance of the model. Experimental results show that the proposed model is better than a single DPCNN model, which further improves the accuracy of text classification tasks.

Analysis and Prediction of Air Quality based on LSTM Neural Network —— Take Beijing Temple of Heaven as an Example

LSTM is an excellent variant model of RNN, which inherits the characteristics of most RNN models and solves the problem of gradient disappearance caused by a gradual reduction in the process of gradient backpropagation. LSTM is very suitable for processing data highly related to time series. This paper selects meteorological elements and air quality data of the temple of heaven in Beijing from 2016 to 2018, analyse the correlation between meteorological elements and air quality, selects index modeling, and designs an air quality prediction model based on LSTM neural network. It is concluded that the model has better accuracy and robustness. Finally, reasonable air pollution control measures are proposed to provide a scientific theoretical basis and new prediction methods for the prevention and control of air pollution.

Low-sample size remote sensing image recognition based on a multihead attention integration network

For a long time, small sample recognition for hyperspectral images has been a popular research topic. It is very difficult for an algorithm to simultaneously satisfy the requirements of feature mining, feature selection and feature integration. The traditional single model has difficulty completing multiple tasks at the same time, ultimately leading to poor small sample recognition results for remote sensing images. This paper proposes a multimodel joint algorithm for deep feature mining based on multiscale convolution (MC) under multihead attention (MA) and deep feature integration based on bidirectional independent recurrent neural networks (BiIndRNNs), MACBINet. First, this paper proposes a multihead attention mechanism that assigns multiple weight coefficients to each feature to better select remote sensing image features; then, it implements the deep mining of features and the retention of multiple deep features through multiscale convolution. Subsequently, it implements contextual semantic information integration for long-sequence features through bidirectional independent recurrent neural networks to avoid the problem of gradient disappearance during training on a small sample of data. Finally, the softmax function is used to perform recognition on three public remote sensing data sets. The experimental results prove that our proposed MACBINet achieves the best results to date for small sample classification.

Research on Situation System Based on Big Data Image -- Taking Road Traffic as an Example

In this paper, based on convolution neural network, object class awareness is carried out, Relu is selected as the activation function, which speeds up the training speed and overcomes the problem of gradient disappearance; using situation awareness technology and database technology, all object classes in the image scene are identified, and its location is specified. This paper takes the road traffic situation awareness system as the research object, and analyzes the information collection, processing and analysis process of the road traffic situation awareness system. CNN, R-CNN, Fast R-CNN and Faster R-CNN are used for vehicle classification and location recognition in road image big data.

Sentiment analysis of film reviews based on CNN-BLSTM-Attention

In order to accurately analyse the emotional tendency of film reviews, help investors make decisions and improve the quality of works, an optimized CNN-BLSTM-Attention sentiment analysis model was designed. The CNN model has a strong ability to capture the local correlation of spatial or temporal structures. The RNN model can either process sequences of any length or capture long-range dependencies, but it is easy to cause the problem of gradient disappearance. The CNN-BLSTM-Attention sentiment analysis model designed in this paper, which combines the advantages of CNN and RNN, is more accurate when being used to analyze the sentiment characteristics of texts. The experimental results show that the accuracy of the CNN-BLSTM-Attention after optimization model is better than that of CNN and RNN models in the experiment, which proves the effectiveness of the analysis method in this paper and can provide some significance for the optimization of related sentiment analysis models.

Real-time micro-expression recognition based on ResNet and atrous convolutions

Based on the depth feature of facial micro-expression recognition, such as by using Convolutional Neural Network, the method of classifying facial micro-expression recognition has been gradually improved. Compared with the traditional feature extraction method, the improved technique can more easily realize real-time application. To perfect the details and extract the fine features of micro-expressions, a new algorithm named MACNN (Multi-scale convolutional neural network model), which combines the kernels of atrous convolutions and automatic face correction, is proposed to improve the feature extraction process of the CNN network. Residual blocks are introduced to solve the gradient disappearance problem and to accelerate convergence during the model training. The model is trained and tested on the micro-expression public data sets CASME and CASME II through real-time detection in real-time application of automatic face correction. The robustness of the model is improved by comparing the loss function schemes. The accuracy of the method is 73%, and the real-time detection occurs at a frame rate of 60 FPS. This method can effectively improve the accuracy of micro-expression detection, meet the real-time requirements, and demonstrate satisfactory robustness and generalizability.

Pulsar candidate selection based on self-normalizing neural networks

Pulsar candidate selection is an important step in the search task of pulsars. The traditional candidate selection is heavily dependent on human inspection. However, the human inspection is a subjective, time consuming, and error-prone process. A modern radio telescopes pulsar survey project can produce totally millions of candidates, so the manual selection becomes extremely difficult and inefficient due to a large number of candidates. Therefore, this study focuses on machine learning developed in recent years. In order to improve the efficiency of pulsar candidate selection, we propose a candidate selection method based on self-normalizing neural networks. This method uses three techniques: self-normalizing neural networks, genetic algorithm and synthetic minority over-sampling technique. The self-normalizing neural networks can improve the identification accuracy by applying deep neural networks to pulsar candidate selection. At the same time, it solves the problem of gradient disappearance and explosion in the training process of deep neural networks by using its self-normalizing property, which greatly accelerates the training process. In addition, in order to eliminate the redundancy of the sample data, we use genetic algorithm to choose sample features of pulsar candidates. The genetic algorithm for feature selection can be summarized into three steps: initializing population, assessing population fitness, and generating new populations. Decoding the individual with the largest fitness value in the last generation population, we can obtain the best subset of features. Due to radio frequency interference or noise, there are a large number of non-pulsar signals in candidates, and only a few real pulsar signals exist there. Aiming at solving the severe class imbalance problem, we use the synthetic minority over-sampling technique to increase the pulsar candidates (minority class) and reduce the imbalance degree of data. By using <i>k</i>-nearest neighbor and linear interpolation to insert a new sample between two minority classes of samples that are close to each other according to certain rules, we can prevent the classifier from becoming biased towards the abundant non-pulsar class (majority class). Experimental results on three pulsar candidate datasets show that the self-normalizing neural network has higher accuracy and faster convergence speed than the traditional artificial neural network in the deep structure, By using the genetic algorithm and synthetic minority over-sampling technique, the selection performance of pulsar candidates can be effectively improved.

An Entire-and-Partial Feature Transfer Learning Approach for Detecting the Frequency of Pest Occurrence

Detecting the frequency of the pest occurrence is always a time consuming and laborious task for agriculture. This paper attempts to solve the problem through the combination of deep learning and pest detection. We propose an entire-and-partial feature transfer learning architecture to perform pest detection, classification and counting tasks which can reach the final goal for detecting the frequency of pest occurrence. In the partial-feature transfer learning, different fine-grained feature map are strengthened by using the weighting scheme of the entire-feature transfer learning. Finally, the cross-layer of the entire-feature network is combined with the multi-scale feature map. The entire-feature transfer learning approach enhances the feature map by creating a shortcut topology for the input and output layers to reduce the gradient disappearance problem which is common to deep networks. The experimental results show that the detection accuracy can be significantly improved and the accuracy can reach 90.2%.

A Wavelet Based Deep Learning Method for Underwater Image Super Resolution Reconstruction

In order to solve the scattering degradation by turbulence and suspended particles in underwater imaging, traditional processing methods including image enhancement, restoration and reconstruction have been continuously researched. But most of them rely on degradation models, and there exist problems of ill-posed. Image super resolution reconstruction based on deep learning has become a hot topic in recent years. In order to further improve the effectiveness and efficiency of deep learning based methods, an improved image super-resolution reconstruction algorithm based on deep convolutional neural network is proposed in this paper. The wavelet basis which can effectively simulate the waveform and characteristics of underwater turbulence is selected to replace the neuron fitting function in order to improve the accuracy and efficiency of the algorithm. An improved dense block structure (IDB) is introduced into the network which can effectively solve the gradient disappearance problem of deep convolutional neural network and improve the training speed at the same time. The method proposed in this paper has been verified in laboratory flume, public data set and real water body. The experimental results show that under the same conditions, the proposed algorithm shows improvements on various evaluation parameters compared with DRFN, VDSR and DRCN method. So it can be concluded that the proposed method can effectively improve the quality of deep learning based reconstruction for imaging in natural water.

Investigation of training performance of convolutional neural networks evolved by genetic algorithms using an activity function

This article presents a study on the training performance of convolutional neural networks (CNN) evolved by genetic algorithms (GA) using an activity function for image recognition. Globally, GA has been considered as one of the most robust search optimization methods in machine learning and artificial intelligent systems. Currently, when CNN is used in 2D image recognition, the ReLU activity function is mostly applied with back propagation (BP) for signal processing and image recognition, because the sigmoid function has a gradient disappearance problem. Although the sigmoid function is good for three-layered neural networks, its performance degrades for multilayer neural networks, especially in BP training. In this study, we also focus on the training performance of an activity function with CNN evolved by GA, especially when the intermediate convolution layers are used. We also evaluate the training accuracy of various activity functions for image recognition with CNN for an automatic driving application using the GA training method.