Neural Network Learning Methods Research Articles

Fault detection in Tennessee Eastman process with temporal deep learning models

Automated early process fault detection and prediction remains a challenging problem in industrial processes. Traditionally it has been done by multivariate statistical analysis of sensor readings and, more recently, with the help of machine learning methods. The quality of machine learning models strongly depends on feature engineering, that in turn heavily relies on expertise of the process engineers and model developers. With the recent advent of deep learning neural network methods and abundance of available sensor data, it became possible to develop advanced approaches to early fault detection and prediction that do not require feature engineering and provide more accurate and timely results.In this paper we investigate a wide range of recurrent and convolutional architectures on the publicly available simulated Tennessee Eastman Process extended TEP dataset for the fault detection in chemical processes. We have selected the best architecture for the task and proposed a novel temporal CNN1D2D architecture that achieves overall better performance on the dataset than any referenced method. We have also proposed to use Generative Adversarial Network GAN to extend and enrich data used in training.

Detecting abusive Instagram comments in Turkish using convolutional Neural network and machine learning methods

Instagram is a free photo-sharing platform where each user has a profile and can upload photos for followers to view, like, and comment. Abusive comments on images can be humiliating and harmful to those who share photos. Developing a comment filter in languages other than English is difficult and time-consuming. This paper proposes a dataset called Abusive Turkish Comments (ATC) to detect abusive Instagram comments in Turkish. It is composed of a large number of Instagram comments posted to tabloid and sports accounts (i.e., 10,528 abusive and 19,826 not-abusive). It is the first public dataset dedicated to detecting abusive Turkish messages, as far as we know. The sentiment annotation has been done in sentence-level by assigning polarity to each comment. The performance of the abusive message detection models was evaluated using several performance metrics: Convolutional Neural Network (CNN), five well-known classifiers (i.e., Naive Bayes, Support Vector Machine, Decision Tree, Random Forest, and Logistic Regression), and two reweighted classifiers (i.e., Adaptive Boosting (AdaBoost), eXtreme Gradient Boosting (XGBoost)) were compared in terms of F1-score, precision, and recall. The results showed that the best performance (i.e., Micro-averaged F1-score: 0.974, Macro-averaged F1-score: 0.973, Kappa-value: 0.946) was yielded by the CNN model on the oversampled ATC dataset. The abusive message detection model proposed in this study can contribute to the development of Turkish comment filters on Instagram. Different model combinations are considered to select the best model that gives better recognition accuracy.

Designing a computerized information processing system to build a movement trajectory of an unmanned aircraft

This paper addresses the issue of developing a computerized system for processing information in the construction of the trajectory of an unmanned aircraft (UAC), a remotely-piloted aviation system (RPAS), or another robotic system. Resolving this task involves the neural network learning algorithms based on the mathematical model of movement. The construction of such a trajectory between two specified destinations has been considered that provides for the possibility of bypassing static and dynamic obstacles. The specified trajectory is divided into several smaller parts. The possibility of restructuring when changing the position of obstacles in space has been considered. A UAC flight control algorithm has been developed, which implies training a neural network for bypassing obstacles of different sizes. To predict the development of the situation when an object moves between two specified points in space, it is proposed to use the Q-Learning algorithm. It has been shown that the smallest number of steps required for moving along a specified trajectory is 18, the largest is 273 steps. In case of distortion during data transmission, the training of the neural network makes it possible to reduce the possibility of collision with obstacles by improving the accuracy and speed of information transfer between the on-board computer and operator. A system of the video support to moving objects was modeled; dependence charts of the normalized frame size at different parameter values were built. Using the charts makes it possible to determine the function of the maneuver intensity. Existing neural network learning methods such as CNN and LSTM were compared. It has been proven that the success rate reaches 74 % when using CNN only, while it amounts to 92 % at the hybrid application of CNN+LSTM. The simulation results have demonstrated the high efficiency of the developed algorithm

An artificial intelligence-based approach to predicting seismic hillslope stability under extreme rainfall events in the vicinity of Wolsong nuclear power plant, South Korea

Rainfall and earthquakes are two significant triggering factors of mass movement. Since the Gyeongju earthquake on 12 September 2016, which took place near the Wolsong nuclear power plant, many concerns have been raised about the threat posed by landslides during intense rainfall. In this study, we developed a new methodological approach to assess the stability of hillslopes at the catchment scale. We applied a geographical information system (GIS)-based pseudo-static model to 10,000 representative sample points by coupling the steady state infiltration corresponding to extreme rainfall and seismic force. Thus, we obtained the factor of safety of the representative sample points and set it as our target variable. The target variable was divided into two subsets: 80% of the data was used to train the model and 20% was reserved for testing purposes. We then applied a deep learning neural network method to incorporate other spatial geo-environmental data such as topographic, hydrologic, soil, forest, and geology, i.e., independent variables that can be used to predict the factor of safety in the catchment scale. The accuracy of the model was assessed using Pearson’s correlation coefficient, which was 0.97 and 0.98 and root mean square error 0.301 and 0.290 in the cases of the training and testing data, respectively. The prediction results indicate that the integration approach produces reliable, accurate landslide susceptibility maps, which may be helpful to researchers working on landslide management strategies.

Integration of well log-derived facies and 3D seismic attributes for seismic facies mapping: A case study from mansuri oil field, SW Iran

The recognition and mapping of rock facies is a pivotal step in reliable reservoir characterization and planning development operations. The present study aimed to present a practical workflow by integrating three-dimensional (3D) seismic attributes and well-log facies analysis to determine the spatial distribution of electrofacies rock types of the Oligo-Miocene Asmari reservoir in Mansuri oilfield located in the southwestern part of Iran. The workflow begins with integrating well logs, core data, and geological information from the selected ‘key wells’ to build an electrofacies model, which could help reservoir facies to be classified and estimated in the cored and uncored wells, respectively. In addition, a group of 3D seismic attribute volumes derived from amplitude, frequency, and time characteristics of the post-stacked 3D seismic data were identified to specify the heterogeneity of the vertical and lateral rock facies within the reservoir. Additionally, the seismic attributes were analyzed using unsupervised and supervised learning techniques to identify seismic facies. The extraction of the seismic facies at multiple-well locations within the 3D classified volumes demonstrated that the supervised neural network learning method, as compared to the unsupervised approach, could more accurately estimate the desired seismic facies. The results of the ultimate rock facies maps of different vertical slices in the Asmari reservoir indicated that electrofacies with good reservoir quality were best developed on the top of the reservoir, mostly with NW-SE elongation at the crest line of the structure. The applied workflow allowed the localities of the buried sand channels to be identified and mapped. Accordingly, seismic facies mapping shed light on the spatial heterogeneity and quality of the reservoir, which could be used to guide the development drilling for optimal recovering of the unswept hydrocarbons.

Learning Rule for a Quantum Neural Network Inspired by Hebbian Learning

Utilizing the enormous potential of quantum computers requires new and practical quantum algorithms. Motivated by the success of machine learning, we investigate the fusion of neural and quantum computing, and propose a learning method for a quantum neural network inspired by the Hebb rule. Based on an analogy between neuron-neuron interactions and qubit-qubit interactions, the proposed quantum learning rule successfully changes the coupling strengths between qubits according to training data. To evaluate the effectiveness and practical use of the method, we apply it to the memorization process of a neuro-inspired quantum associative memory model. Our numerical simulation results indicate that the proposed quantum versions of the Hebb and anti-Hebb rules improve the learning performance. Furthermore, we confirm that the probability of retrieving a target pattern from multiple learned patterns is sufficiently high.

Review on Convolutional Neural Networks (CNN) in vegetation remote sensing

Identifying and characterizing vascular plants in time and space is required in various disciplines, e.g. in forestry, conservation and agriculture. Remote sensing emerged as a key technology revealing both spatial and temporal vegetation patterns. Harnessing the ever growing streams of remote sensing data for the increasing demands on vegetation assessments and monitoring requires efficient, accurate and flexible methods for data analysis. In this respect, the use of deep learning methods is trend-setting, enabling high predictive accuracy, while learning the relevant data features independently in an end-to-end fashion. Very recently, a series of studies have demonstrated that the deep learning method of Convolutional Neural Networks (CNN) is very effective to represent spatial patterns enabling to extract a wide array of vegetation properties from remote sensing imagery. This review introduces the principles of CNN and distils why they are particularly suitable for vegetation remote sensing. The main part synthesizes current trends and developments, including considerations about spectral resolution, spatial grain, different sensors types, modes of reference data generation, sources of existing reference data, as well as CNN approaches and architectures. The literature review showed that CNN can be applied to various problems, including the detection of individual plants or the pixel-wise segmentation of vegetation classes, while numerous studies have evinced that CNN outperform shallow machine learning methods. Several studies suggest that the ability of CNN to exploit spatial patterns particularly facilitates the value of very high spatial resolution data. The modularity in the common deep learning frameworks allows a high flexibility for the adaptation of architectures, whereby especially multi-modal or multi-temporal applications can benefit. An increasing availability of techniques for visualizing features learned by CNNs will not only contribute to interpret but to learn from such models and improve our understanding of remotely sensed signals of vegetation. Although CNN has not been around for long, it seems obvious that they will usher in a new era of vegetation remote sensing.

Learning Efficient Binarized Object Detectors With Information Compression.

In this paper, we propose a binarized neural network learning method (BiDet) for efficient object detection. Conventional network binarization methods directly quantize the weights and activations in one-stage or two-stage detectors with constrained representational capacity, so that the information redundancy in the networks causes numerous false positives and degrades the performance significantly. On the contrary, our BiDet fully utilizes the representational capacity of the binary neural networks by redundancy removal, through which the detection precision is enhanced with alleviated false positives. Specifically, we generalize the information bottleneck (IB) principle to object detection, where the amount of information in the high-level feature maps is constrained and the mutual information between the feature maps and object detection is maximized. Meanwhile, we learn sparse object priors so that the posteriors are concentrated on informative detection prediction with false positive elimination. Since BiDet employs a fixed IB trade-off to balance the total and relative information contained in the high-level feature maps, the information compression leads to ineffective utilization of the network capacity or insufficient redundancy removal for input in different complexity. To address this, we further present binary neural networks with automatic information compression (AutoBiDet) to automatically adjust the IB trade-off for each input according to the complexity. Moreover, we further propose the class-aware sparse object priors by assigning different sparsity to objects in various classes, so that the false positives are alleviated more effectively without recall decrease. Extensive experiments on the PASCAL VOC and COCO datasets show that our BiDet and AutoBiDet outperform the state-of-the-art binarized object detectors by a sizable margin.

Remote Sensing Estimation of Chlorophyll-A in Case-II Waters of Coastal Areas: Three-Band Model Versus Genetic Algorithm–Artificial Neural Networks Model

Chlorophyll-a (Chl-a), an important indicator of phytoplankton biomass and eutrophication, is sensitive to water constitutes and optical characteristics. An integrated machine learning method of genetic algorithm and artificial neural networks (GA–ANN) was developed to retrieve the concentration of Chl-a. In situ spectra and simultaneous water quality parameters of 107 samples from two reservoirs (Res) and coastal waters (CW) were used to calibrate GA–ANN and three-band models (TBM) for comparison of Chl-a estimation. Both GA–ANN and TBM methods perform well for the joint dataset (WGD) of Res and CW with the R 2 exceeding 0.90, and the root mean square error (RMSE) of corresponding validation ( N = 35) are 4.40 and 5.23 μ g/L, respectively. Similarly, for independent dataset of Res ( N = 45), GA–ANN and TBM methods show robust performance: the R 2 values are 0.87 and 0.80, respectively; and the corresponding RMSE values are 7.79 and 7.73 μ g/L, respectively. For CW dataset ( N = 62), the R 2 values of two methods are 0.81 and 0.62, respectively; and the corresponding RMSE values are 0.79 and 1.32 μ g/L, respectively. When the GA–ANN and TBM models were applied to retrieve Chl-a concentration from the calibrated Sentinel 2 MSI reflectance data in two Res on October 20, 2019, however, the validated results of MSI-derived Chl-a concentrations using quasi-synchronous in situ data ( N = 36) indicated that the GA–ANN model outperforms TBM with higher R 2 value (0.91 vs. 0.26) and smaller RMSE (4.41 vs. 13.85 μ g/L) and mean absolute errors (3.40 vs. 11.87 μ g/L) values. Although TBM has obvious overestimation of Chl-a concentration when applied to remote sensing image, we still thought that both GA–ANN and TBM are useful methods for Chl-a estimation in case-II waters, and GA–ANN performs marginally better with less deviation to measured Chl-a for multispectral remote sensing data. The ratio of TSS to Chl-a, experimental measurements, abundance of sampling points, and Chl-a concentration range are several important factors affecting the accuracy and robustness of GA–ANN and TBM methods.

Development of machine learning techniques to enhance turbulence models

The implementation of the machine learning methods of convolutional neural networks to enhance RANS closure models is presented. The RANS models are not universal and accurate, however they are computationally affordable. Finding a way to improve the model predictability will be an advantage. For this, machine learning algorithms based on available high-fidelity data sets for canonical flow cases obtained from DNS and measurements can be helpful. The application of these algorithms for a fully-developed turbulent channel flows with periodic hills, in a square duct and for other cases is considered.

Non-Profiled Side-Channel Attack Based on Deep Learning Using Picture Trace

Over the years, deep learning algorithms have advanced a lot and any innovation in the algorithms are demonstrated and benchmarked for image classification. Several other field including side-channel analysis (SCA) have recently adopted deep learning with great success. In SCA, the deep learning algorithms are typically working with 1-dimensional (1-D) data. In this work, we propose a unique method to improve deep learning based side-channel analysis by converting the measurements from raw-trace of 1-dimension data based on float or byte data into picture-formatted trace that has information based on the data position. We demonstrate why “Picturization” is more suitable for deep learning and compare how input and hidden layers interact for each raw (1-D) and picture form. As one potential application, we use a Binarized Neural Network (BNN) learning method that relies on a BNN's natural properties to improve variables. In addition, we propose a novel criterion for attack success or failure based on statistical confidence level rather than determination of a correct key using a ranking system.

DeepDeSpy: A Deep Learning-Based Wireless Spy Camera Detection System

Spy cameras planted in various private places, such as motels, hotels, homestays ( i.e., Airbnb), and restrooms, have raised immense privacy concerns. Wi-Fi spy cameras are used extensively by various adversaries because of easy installability, followed by size reduction. To prevent invasions of privacy, most studies have detected wireless cameras based on video traffic analysis and require additional synchronous data from external sensors or stimulus hardware to confirm the user’s motion. Such supplements make the users uncomfortable, requiring extra effort and time for setting. This paper proposes an effective spy camera detection system called DeepDeSpy to detect the recording of a spy camera with no effort from the user. The core idea is using the channel state information (CSI) and the network traffic from the camera to detect whether the wireless camera records the movements of the user. The CSI signal is prone to motion, and detecting motion from an enormous amount of CSI data in real-time is challenging. This was handled by leveraging the convolutional neural network (CNN) and bidirectional long short-term memory (BiLSTM) deep learning methods. Such synergistic CNN and BiLSTM deep learning models enable instant and accurate detection by automatically extracting meaningful features from the sequential raw CSI data. The feasibility of DeepDeSpy was verified by implementing it on both a PC and a smartphone and evaluating it in real-life scenarios (e.g., various room sizes and user physical activities). The average accuracy achieved in different real-life settings was approximately 96%, reaching 98.9% with intensive physical activity in the large-size room. Moreover, the ability to achieve instant detection on a smartphone within only a one-second response time makes it workable for real-time applications.

Research on Bone Age Automatic Judgment Algorithm Based on Deep Learning and Hand X-ray Image

Objective: To learn the depth of paper using feature extraction, combined with critical areas of heat syndrome and related information, X-ray image of hand to analyze bone age children. Methods: The thesis of the X-ray image data preprocessing left hand, the use of depth of depth neural network learning methods, combined with clinical data skeletal age evaluation model to evaluate the effectiveness of the test model. Results: X-ray image of hand artificial feature extraction, combined SVM classification, automatic assessment of skeletal age. The method of automatic assessment of bone age SVM-based feature primarily artificial, SIFT features extracted image, LBP features, characteristics of GCLM, these features are combined, and then used to train the SVM, have some ability to automatically assess bone age assessment based on SVM. Conclusion: This topic X-ray image based on the hand bones, computer vision, machine learning to extract the relevant methods, pretreatment and segmentation of X-ray images of the hand bones, characterized by automatic assessment of bone age, lack the core image of the sample problem.

Hardware-Based Hopfield Neuromorphic Computing for Fall Detection.

With the popularity of smart wearable systems, sensor signal processing poses more challenges to machine learning in embedded scenarios. For example, traditional machine-learning methods for data classification, especially in real time, are computationally intensive. The deployment of Artificial Intelligence algorithms on embedded hardware for fast data classification and accurate fall detection poses a huge challenge in achieving power-efficient embedded systems. Therefore, by exploiting the associative memory feature of Hopfield Neural Network, a hardware module has been designed to simulate the Neural Network algorithm which uses sensor data integration and data classification for recognizing the fall. By adopting the Hebbian learning method for training neural networks, weights of human activity features are obtained and implemented/embedded into the hardware design. Here, the neural network weight of fall activity is achieved through data preprocessing, and then the weight is mapped to the amplification factor setting in the hardware. The designs are checked with validation scenarios, and the experiment is completed with a Hopfield neural network in the analog module. Through simulations, the classification accuracy of the fall data reached 88.9% which compares well with some other results achieved by the software-based machine-learning algorithms, which verify the feasibility of our hardware design. The designed system performs the complex signal calculations of the hardware’s feedback signal, replacing the software-based method. A straightforward circuit design is used to meet the weight setting from the Hopfield neural network, which is maximizing the reusability and flexibility of the circuit design.

Lane departure warning systems and lane line detection methods based on image processing and semantic segmentation: A review

Abstract Recently, the development and application of lane line departure warning systems have been in the market. For any of the systems, the key part of lane line tracking, lane line identification, or lane line departure warning is whether it can accurately and quickly detect lane lines. Since 1990s, they have been studied and implemented for the situations defined by the good viewing conditions and the clear lane markings on road. After then, the accuracy for particular situations, the robustness for a wide range of scenarios, time efficiency and integration into higher-order tasks define visual lane line detection and tracking as a continuing research subject. At present, these kinds of lane marking line detection methods based on machine vision and image processing can be divided into two categories: the traditional image processing and semantic segmentation (includes deep learning) methods. The former mainly involves feature-based and model-based steps, and which can be classified into similarity- and discontinuity-based ones; and the model-based step includes different parametric straight line, curve or pattern models. The semantic segmentation includes different machine learning, neural network and deep learning methods, which is the new trend for the research and application of lane line departure warning systems. This paper describes and analyzes the lane line departure warning systems, image processing algorithms and semantic segmentation methods for lane line detection.

DeepSense—Deep neural network framework to improve the network lifetime of IoT‐MANETs

SummaryWith the advancement of Internet of Things (IoT), the devices are allowed to interact with other networks like mobile ad hoc network (MANET). The MANET‐IoT systems often undergo energy balancing problem between the sensor nodes, whereas the MANETs operate on mobile sensor nodes. Hence, proper utilization of battery power is required to maintain the network connectivity during a multi‐hop transmission. In this paper, we propose a DeepSense IoT‐MANET framework that effectively routes the packets from the IoT nodes via mobile sensor nodes in MANETs. The routings between the MANETs are organized by DeepSense interconnected with deep neural network (DNN) learning methods. The performance of the DeepSense DNN method is evaluated against various network metrics to evaluate the efficacy of the model.

Matrix Expression of Convolution and Its Generalized Continuous Form

In this paper, we consider the matrix expression of convolution, and its generalized continuous form. The matrix expression of convolution is effectively applied in convolutional neural networks, and in this study, we correlate the concept of convolution in mathematics to that in convolutional neural network. Of course, convolution is a main process of deep learning, the learning method of deep neural networks, as a core technology. In addition to this, the generalized continuous form of convolution has been expressed as a new variant of Laplace-type transform that, encompasses almost all existing integral transforms. Finally, we would, in this paper, like to describe the theoretical contents as detailed as possible so that the paper may be self-contained.

Consistent constraint-based video-level learning for action recognition

This paper proposes a new neural network learning method to improve the performance for action recognition in video. Most human action recognition methods use a clip-level training strategy, which divides the video into multiple clips and trains the feature learning network by minimizing the loss function of clip classification. The video category is predicted by the voting of clips from the same video. In order to obtain more effective action feature, a new video-level feature learning method is proposed to train 3D CNN to boost the action recognition performance. Different with clip-level training which uses clips as input, video-level learning network uses the entire video as the input. Consistent constraint loss is defined to minimize the distance between clips of the same video in voting space. Further, a video-level loss function is defined to compute the video classification error. The experimental results show that the proposed video-level training is a more effective action feature learning approach compared with the clip-level training. And this paper has achieved the state-of-the-art performance on UCF101 and HMDB51 datasets without using pre-trained models of other large-scale datasets. Our code and final model are available at https://github.com/hqu-cst-mmc/VLL.

Real-time video based emotion recognition using convolutional neural network and transfer learning

Background/Objectives: The deep learning approaches have paved their way to construct various artificial intelligence products and the proposed system uses a convolutional neural network for detecting real-time emotions of mankind. The objective of the study is to develop a real-time application for emotion recognition using convolutional neural network and transfer learning methods. Methods/Statistical analysis: The proposed system considers happy, normal and surprised categories of emotions. The system consists of four major steps: dataset collection, training, validation, and real-time testing. The dataset is comprised of face images containing emotions such as happy,normal and surprised in the form of video frames. The face and mouth regions are detected using the Haar-Based Cascade classifier at 20 frames per second. Findings: The convolutional neural network (CNN) is trained using mouth images and the pre-trained models VGG16 and VGG19 are trained with face images. The trained model is used to detect the emotions in the live webcam video. The experimental results show that the CNN model trained using mouth images gives an accuracy of 85.71% and the pre-trained models trained with face images using transfer learning method achieves an accuracy of 77.78%. The proposed system using CNN outperforms the pre-trained models for recognizing the emotions in real-time video. Novelty/Applications:The proposed system is entirely based on the mouth region video frames and the real-time emotion recognition system is developed. This work can detect the three emotions in an unconstrained laboratory environment. Keywords: Convolutional neural network; mouth detection; pre-trained models; real-time emotion recognition; transfer learning

Machine Learning Approach to Analyse Ensemble Models and Neural Network Model for E-Commerce Application

Objectives: The main objective of this study is to compare the performance evaluation of ensemble based methods and neural network learning on various combinations of unigram, bigram, and trigram feature vector along with feature selection (IG) and feature reduction (PCA) for sentiment classification of movie reviews. Methods: Bagging and Adaboost are the techniques used in ensemble learning to learn the sentiment classifier to get better classification accuracy, using SVM, NB as a core learner for different models of attribute vectors. The classification results of the ensemble approach are compared with neural network learning for classification of movie reviews. Among the ensemble methods, AdaBoost with base learner SVM outperforms in classifying attribute vectors for model m-iii. The backpropagation algorithm is used to improve classification accuracy in the neural network learning and IG and PCA are used in sentiment classification to reduce the feature length and training time. Findings:The classification results of ensemble based approach are compared with neural network learning. Between the two ensemble based methods, Adaboost + SVM outperform in classifying the sentiment of movie reviews for m-iii feature vector. IG and PCA are used in sentiment classification in order to reduce the feature length. Between the IG and PCA methods, IG performs better than PCA. Among IG+Adaboost+SVM and neural network learning methods, IG+Adaboost+SVM performs better than neural network learning. Improvement: In our application, we are using the ensemble based methods and neural network learning, these methods are compared and analyzed the performance for various levels of feature vectors. A classification algorithm may be designed to analyze the performance with other neural network methods. Keywords: Machine learning; sentiment classification; bagging; AdaBoost; ensemble learning; back propagation neural network; feature selection; movie review