Discriminant Neural Networks Research Articles

Development of deep learning applications in FPGA-based fusion diagnostics using IRIO-OpenCL and NDS

Many of the signals that are relevant to fusion science come from 1D signals or time-series. In this field, the resulting Neural Networks are much simpler than the more mainstream vision-based neural networks. A significant reduction in both dimension and complexity make them suitable to be synthesized in FPGAs. We have developed new features for the IRIO-OpenCL platform to support this technology for fusion problems. The work presented analyzes the feasibility of such diagnostics use cases and how they can be integrated with the help of OpenCL technology. The development and testing platform consists of an MTCA.4 system with an AMC module integrating an Intel Arria 10 FPGA. An ADC connected using the FMC interface samples the analog signals passed to the OpenCL processing kernels. By using OpenCL, the FPGA kernels communicate with the host machine in a standardized way. This brings two main advantages. First, this is an ideal prototyping framework. Second, once a solution is final, the FPGA kernels are integrated into the control system (EPICS) using the IRIO-OpenCL layer, which has been developed with Nominal Device Support (NDSv3). Finally, we present the results of the optimizations to the kernels of an application example based on a neutron/gamma discrimination Neural Network, which achieves up to a classification rate of 1.3 MEvents/s.

PCA-RF model for Dendrolimus punctatus Walker damage detection

At the present stage, the effective coupling information of “ground-space” is still a fundamental way to detect forest pest damage rapidly and accurately in remote sensing. It is of great significance to explore and construct a detection model that can comprehensively and effectively utilize ground microscopic features and remote sensing pixel information. Taking Dendrolimus punctatus Walker damage as the study object, the host characteristics and the differences with healthy pine forest are analyzed from the aspects of leaf volume, greenness, moisture, forest form and spectrum. Four experimental sites of Sanming City, Jiangle County, Sha County and Yanping District in Nanping City, Fujian, China are set as the experimental areas, and two forest stand features of LAI and SEL are measured, the remote sensing indicators of NDVI, WET and B2, B3, B4 are calculated or extracted. The PCA-RF detection model is constructed with pest levels of non-damage, mild damage, moderate damage and severe damage as dependent variables. This model reduces the dimensions by converting the initial variables into several principal components and making the principal components input to the random forest. The first three principal components of PCA can better replace the information of the original seven characteristic indicators, thereby reducing the seven-dimensional information to three dimensions. One hundred eighty-two samples are randomly divided into the training set and test set for the five repeats. The detection accuracy, Kappa coefficient, ROC are selected to evaluate the pest damage detection effects of PCA-RF model, and compare with Fisher discriminant analysis and BP neural network. The results show that the detection accuracy, Kappa coefficient and AUC of PCA-RF model in the training set are not as good as those of FDA and BPNN, but the detection accuracy and Kappa coefficient in the test set are significantly better than the other two algorithms, and the AUC value is also significantly higher than BNPP. This study proves that PCA-RF model simplifies the complex problem, inherits the advantages of random forest, has very strong generalization ability and robustness, and can be used as an effective solution for forest pest and disease detection.

Identification of the raw and processed Crataegi Fructus based on the electronic nose coupled with chemometric methods

Crataegi Fructus (CF) is widely used as a medicinal and edible material around the world. Currently, different types of processed CF products are commonly found in the market. Quality evaluation of them mainly relies on chemical content determination, which is time and money consuming. To rapidly and nondestructively discriminate different types of processed CF products, an electronic nose coupled with chemometrics was developed. The odour detection method of CF was first established by single-factor investigation. Then, the sensor array was optimised by a stepwise discriminant analysis (SDA) and analysis of variance (ANOVA). Based on the best-optimised sensor array, the digital and mode standard were established, realizing the odour quality control of samples. Meanwhile, mathematical prediction models including the discriminant formula and back-propagation neural network (BPNN) model exhibited good evaluation with a high accuracy rate. These results suggest that the developed electronic nose system could be an alternative way for evaluating the odour of different types of processed CF products.

Bidirectional Attention for Text-Dependent Speaker Verification.

Automatic speaker verification provides a flexible and effective way for biometric authentication. Previous deep learning-based methods have demonstrated promising results, whereas a few problems still require better solutions. In prior works examining speaker discriminative neural networks, the speaker representation of the target speaker is regarded as a fixed one when comparing with utterances from different speakers, and the joint information between enrollment and evaluation utterances is ignored. In this paper, we propose to combine CNN-based feature learning with a bidirectional attention mechanism to achieve better performance with only one enrollment utterance. The evaluation-enrollment joint information is exploited to provide interactive features through bidirectional attention. In addition, we introduce one individual cost function to identify the phonetic contents, which contributes to calculating the attention score more specifically. These interactive features are complementary to the constant ones, which are extracted from individual speakers separately and do not vary with the evaluation utterances. The proposed method archived a competitive equal error rate of 6.26% on the internal “DAN DAN NI HAO” benchmark dataset with 1250 utterances and outperformed various baseline methods, including the traditional i-vector/PLDA, d-vector, self-attention, and sequence-to-sequence attention models.

A Robust Deep-Learning-Enabled Trust-Boundary Protection for Adversarial Industrial IoT Environment

In recent years, trust-boundary protection has become a challenging problem in Industrial Internet of Things (IIoT) environments. Trust boundaries separate IIoT processes and data stores in different groups based on user access privilege. Points where dataflow intersects with the trust boundary are becoming entry points for attackers. Attackers use various model skewing and intelligent techniques to generate adversarial/noisy examples that are indistinguishable from natural data. Many of the existing machine-learning (ML)-based approaches attempt to circumvent this problem. However, owing to an extremely large attack surface in the IIoT network, capturing a true distribution during training is difficult. The standard generative adversarial network (GAN) commonly generates adversarial examples for training using randomly sampled noise. However, the distribution of noisy inputs of GAN largely differs from actual distribution of data in IIoT networks and shows less robustness against adversarial attacks. Therefore, in this article, we propose a downsampler-encoder-based cooperative data generator that is trained using an algorithm to ensure better capture of the actual distribution of attack models for the large IIoT attack surface. The proposed downsampler-based data generator is alternatively updated and verified during training using a deep neural network discriminator to ensure robustness. This guarantees the performance of the generator against input sets with a high noise level at time of training and testing. Various experiments are conducted on a real IIoT testbed data set. Experimental results show that the proposed approach outperforms conventional deep learning and other ML techniques in terms of robustness against adversarial/noisy examples in the IIoT environment.

Structure-preserving shape completion of 3D point clouds with generative adversarial network

Due to the difficulty in maintaining the fine structures of 3D point cloud in shape completion, this study, with the help of the generative adversarial network framework, proposes a novel neural network for automatically repairing and completing the 3D shape of point clouds. This network consists of a generator and a discriminator. The generator of the proposed neural network adopts an encoder-decoder structure and takes the missing 3D point cloud shape data as the input. Firstly, it aligns the sampling point position and feature information of the input point cloud data by the input transform and feature transform. Then the weighted shared multi-layer perceptron extracts the local shape features for each sampling point and also extracts its feature codewords using the maximum pool layer and multi-layer perceptron coding. Secondly, it adds the feature codewords of sampling points with the grid coordinate data, and the decoder converts the grid data into the missing data of the underlying point cloud using two successive three-layer perceptron folding operations. Finally, it merges the missing completion data and the input data to get the complete 3D point cloud shape. Meanwhile, the proposed neural network discriminator receives the real and the completed point cloud data generated by the generator. The same encoder structure as the generator is also adopted to distinguish the true or false of the point cloud data, while the classification results are a feedback for optimizing the generator. Also, the generator will generate the “real” point cloud shape data. Experimental results illustrate that, for both the dense and sparse incomplete point cloud data, the proposed method effectively maintains the fine structures of the input point clouds while repairing the missing part of the underlying shapes.

CECAV-DNN: Collective Ensemble Comparison and Visualization using Deep Neural Networks

We propose a deep learning approach to collectively compare two or multiple ensembles, each of which is a collection of simulation outputs. The purpose of collective comparison is to help scientists understand differences between simulation models by comparing their ensemble simulation outputs. However, the collective comparison is non-trivial because the spatiotemporal distributions of ensemble simulation outputs reside in a very high dimensional space. To this end, we choose to train a deep discriminative neural network to measure the dissimilarity between two given ensembles, and to identify when and where the two ensembles are different. We also design and develop a visualization system to help users understand the collective comparison results based on the discriminative network. We demonstrate the effectiveness of our approach with two real-world applications, including the ensemble comparison of the community atmosphere model (CAM) and the rapid radiative transfer model for general circulation models (RRTMG) for climate research, and the comparison of computational fluid dynamics (CFD) ensembles with different spatial resolutions.

Discriminative neural network pruning in a multiclass environment: A case study in spoken emotion recognition

Deep learning has become one of the most widely accepted paradigms regarding machine learning. It focuses on the use of hierarchical data models and builds upon the notion that in order to learn about high level data representations, a better understanding of intermediate level representation is needed. Restricted Boltzmann Machines and deep belief networks are two main types of deep learning algorithms commonly used in a wide array of classification and pattern recognition tasks. Examples of these tasks are natural language recognition, neuroimaging studies, forecasting time series, parametric voice synthesis, and speech emotion recognition among others. Recent machine learning studies suggest that deep learning networks can help map feature problems into a more advantageous position, hence improving the classification process. However, selecting a suitable Deep learning architecture in response to a specific problem can be difficult. In this study, we intend to investigate whether discriminative measures, such as Anova, Pearsonâs Correlation, Fisher score, Gain ratio, ReliefF, OneR among others, could offer pointers to identify useful neural nods in a Deep learning network. This is due to the fact that normally not all hidden neurons provide insightful information for a classification task. Our approach consists in using some of these discriminative measures to rank the hidden neurons based on their output values, and then prune them in accordance to their position within said ranking. Our results indicate that this approach is also helpful in multiclass classification problems and the pruning process seems to have a positive effect in diminishing the resulting error rate.

Classification of rare traffic signs

The paper studies the possibility of using neural networks for the classification of objects that are few or absent at all in the training set. The task is illustrated by the example of classification of rare traffic signs. We consider neural networks trained using a contrastive loss function and its modifications, also we use different methods for generating synthetic samples for classification problems. As a basic method, the indexing of classes using neural network features is used. A comparison is made of classifiers trained with three different types of synthetic samples and their mixtures with real data. We propose a method of classification of rare traffic signs using a neural network discriminator of rare and frequent signs. The experimental evaluation shows that the proposed method allows rare traffic signs to be classified without significant loss of frequent sign classification quality.

Discriminative Neural Network for Coronary Heart Disease Detection

Cardiovascular disease is one of the commonest diseases and main causes of death over the world. As the major type of cardiovascular diseases, correct and timely diagnosis of coronary heart disease (CHD) is very essential. Traditional back-propagation (BP) neural network aims to train a multilayer feedforward neural network which transforms data into the feature space to learn good decision boundaries. However, the performance of BP neural network tends to deteriorate when dealing with complexity medical diagnostic tasks. To improve the detection of CHD, this study proposes a discriminative neural network called DNN. DNN explores discriminative information by maximizing the difference of the compactness within each class and separability between different classes. DNN integrates the discriminative information into the framework of BP neural network, and can be easily implemented by the existing neural network software. Experimental results on Z-Alizadeh Sani dataset show that DNN achieves satisfactory performance in sensitivity, specificity, accuracy and receiver operating characteristic (ROC) curve.

Deep Clustering: On the Link Between Discriminative Models and K-Means.

In the context of recent deep clustering studies, discriminative models dominate the literature and report the most competitive performances. These models learn a deep discriminative neural network classifier in which the labels are latent. Typically, they use multinomial logistic regression posteriors and parameter regularization, as is very common in supervised learning. It is generally acknowledged that discriminative objective functions (e.g., those based on the mutual information or the KL divergence) are more flexible than generative approaches (e.g., K-means) in the sense that they make fewer assumptions about the data distributions and, typically, yield much better unsupervised deep learning results. On the surface, several recent discriminative models may seem unrelated to K-means. This study shows that these models are, in fact, equivalent to K-means under mild conditions and common posterior models and parameter regularization. We prove that, for the commonly used logistic regression posteriors, maximizing the L2 regularized mutual information via an approximate alternating direction method (ADM) is equivalent to minimizing a soft and regularized K-means loss. Our theoretical analysis not only connects directly several recent state-of-the-art discriminative models to K-means, but also leads to a new soft and regularized deep K-means algorithm, which yields competitive performance on several image clustering benchmarks.

Analysis of credit risk faced by public companies in Brazil: an approach based on discriminant analysis, logistic regression and artificial neural networks

The aims of the present article are to identify the economic-financial indicators that best characterize Brazilian public companies through credit-granting analysis and to assess the most accurate techniques used to forecast business bankruptcy. Discriminant analysis, logistic regression and neural networks were the most used methods to predict insolvency. The sample comprised 121 companies from different sectors, 70 of them solvent and 51 insolvent. The conducted analyses were based on 35 economic-financial indicators. Need of working capital for net income, liquidity thermometer, return on equity, net margin, debt breakdown and equity on assets were the most relevant economic-financial indicators. Neural networks recorded the best accuracy and the Receiver Operating Characteristic Curves (ROC curve) corroborated this outcome.

Hallucinating Unaligned Face Images by Multiscale Transformative Discriminative Networks

Conventional face hallucination methods heavily rely on accurate alignment of low-resolution (LR) faces before upsampling them. Misalignment often leads to deficient results and unnatural artifacts for large upscaling factors. However, due to the diverse range of poses and different facial expressions, aligning an LR input image, in particular when it is tiny, is severely difficult. In addition, when the resolutions of LR input images vary, previous deep neural network based face hallucination methods require the interocular distances of input face images to be similar to the ones in the training datasets. Downsampling LR input faces to a required resolution will lose high-frequency information of the original input images. This may lead to suboptimal super-resolution performance for the state-of-the-art face hallucination networks. To overcome these challenges, we present an end-to-end multiscale transformative discriminative neural network devised for super-resolving unaligned and very small face images of different resolutions ranging from 16 $$\times $$ 16 to 32 $$\times $$ 32 pixels in a unified framework. Our proposed network embeds spatial transformation layers to allow local receptive fields to line-up with similar spatial supports, thus obtaining a better mapping between LR and HR facial patterns. Furthermore, we incorporate a class-specific loss designed to classify upright realistic faces in our objective through a successive discriminative network to improve the alignment and upsampling performance with semantic information. Extensive experiments on a large face dataset show that the proposed method significantly outperforms the state-of-the-art.

Digital pulse processing algorithm for neutron and gamma rays discrimination

This paper presents a proposed algorithm for discriminating neutron and gamma radiation events. In the proposed algorithm, discriminating features are extracted from the input radiation event. These features are extracted using charge integration, matched filtering, and Discrete Wavelet Transform (DWT) methods. The extracted features are then fed into the discriminator which is an Artificial Neural Network (ANN) discriminator or a Support Vector Machine (SVM) discriminator. The obtained results prove that the proposed algorithms can be used efficiently for the neutron and gamma ray discrimination purpose and that the DWT based method achieves the highest discrimination rates. For discriminators, the SVM discriminator achieves better performance with a slightly shorter processing time compared to the ANN discriminator.

Learning from Web Data Using Adversarial Discriminative Neural Networks for Fine-Grained Classification

Fine-grained classification is absorbed in recognizing the subordinate categories of one field, which need a large number of labeled images, while it is expensive to label these images. Utilizing web data has been an attractive option to meet the demands of training data for convolutional neural networks (CNNs), especially when the well-labeled data is not enough. However, directly training on such easily obtained images often leads to unsatisfactory performance due to factors such as noisy labels. This has been conventionally addressed by reducing the noise level of web data. In this paper, we take a fundamentally different view and propose an adversarial discriminative loss to advocate representation coherence between standard and web data. This is further encapsulated in a simple, scalable and end-to-end trainable multi-task learning framework. We experiment on three public datasets using large-scale web data to evaluate the effectiveness and generalizability of the proposed approach. Extensive experiments demonstrate that our approach performs favorably against the state-of-the-art methods.

Learning and inference on generative adversarial quantum circuits

Quantum mechanics is inherently probabilistic in light of Born's rule. Using quantum circuits as probabilistic generative models for classical data exploits their superior expressibility and efficient direct sampling ability. However, training of quantum circuits can be more challenging compared to classical neural networks due to lack of efficient differentiable learning algorithm. We devise an adversarial quantum-classical hybrid training scheme via coupling a quantum circuit generator and a classical neural network discriminator together. After training, the quantum circuit generative model can infer missing data with quadratic speed up via amplitude amplification. We numerically simulate the learning and inference of generative adversarial quantum circuit using the prototypical Bars-and-Stripes dataset. Generative adversarial quantum circuits is a fresh approach to machine learning which may enjoy the practically useful quantum advantage on near-term quantum devices.

Content-Aware Convolutional Neural Network for In-Loop Filtering in High Efficiency Video Coding.

Recently, convolutional neural network (CNN) has attracted tremendous attention and has achieved great success in many image processing tasks. In this paper, we focus on CNN technology combined with image restoration to facilitate video coding performance and propose the content-aware CNN based in-loop filtering for high-efficiency video coding (HEVC). In particular, we quantitatively analyze the structure of the proposed CNN model from multiple dimensions to make the model interpretable and optimal for CNN-based loop filtering. More specifically, each coding tree unit (CTU) is treated as an independent region for processing, such that the proposed content-aware multimodel filtering mechanism is realized by the restoration of different regions with different CNN models under the guidance of the discriminative network. To adapt the image content, the discriminative neural network is learned to analyze the content characteristics of each region for the adaptive selection of the deep learning model. The CTU level control is also enabled in the sense of rate-distortion optimization. To learn the CNN model, an iterative training method is proposed by simultaneously labeling filter categories at the CTU level and fine-tuning the CNN model parameters. The CNN based in-loop filter is implemented after sample adaptive offset in HEVC, and extensive experiments show that the proposed approach significantly improves the coding performance and achieves up to 10.0% bit-rate reduction. On average, 4.1%, 6.0%, 4.7%, and 6.0% bit-rate reduction can be obtained under all intra, low delay, low delay P, and random access configurations, respectively.

Detection and diagnosis of dilated cardiomyopathy from the left ventricular parameters in echocardiogram sequences

The heart has a complicated anatomy and is in constant movement. The cardiologist use echocardiogram to visualise the anatomy and its movement. It is difficult for the cardiologist to prognosticate or affirm the diseases like heart muscle damage, valvular problems, etc. due to presence of less information in echocardiograms. In this paper a system is proposed which automatically segments the left ventricle from the given echocardiogram video sequences using the combination of fuzzy C-means clustering and morphological operations and from which the left ventricle parameters and shape features are evoked. These features are then employed to linear discriminant analysis, K-nearest neighbour and Hopfield neural network to determine whether the heart is normal or affected with DCM. With LV parameters evaluated and shape features extracted it was found that HNN was able to model normal and abnormal hearts very well with an accuracy of 88% compared to LDA and K-NN.

Opportunistic Networks Link Prediction Method Based on Bayesian Recurrent Neural Network

The goal of link prediction is to estimate the possibility of future links among nodes using known network information and the attributes of the nodes. According to the time-varying characteristics and the node’s mobility of opportunistic networks, this paper proposes a novel link prediction method based on the Bayesian recurrent neural network (BRNN-LP) framework. The time series data of a dynamic opportunistic networks is sliced into snapshots in which there exist the correlation information and spatial location information. A vector of a snapshot is constructed based on such information, which represents the link information. Then, the vectors of multiple network snapshots constitute a spatiotemporal vector sequence. Benefiting from the BRNN’s ability of extracting the features of time series data, the correlation between spatiotemporal vector sequence and node connection states is learned, and the law of the link evolution is captured to predict future links. The results on the MIT Reality dataset show that compared with methods such as the similarity-based indices, the support vector classifier, linear discriminant analysis and recurrent neural network, the proposed prediction method is more accurate and stable.

Predictive-Maintenance Approach Uses Machine Learning To Increase Efficiency

This article, written by JPT Technology Editor Chris Carpenter, contains highlights of paper OTC 28990, “Increasing Production Efficiency Through Compressor-Failure Predictive Analytics Using Machine Learning,” by D. Pandya, A. Srivastava, A. Doherty, S. Sundareshwar, C. Needham, A. Chaudry, and S. Krishnalyer, Shell, prepared for the 2018 Offshore Technology Conference, Houston, 30 April–3 May. The paper has not been peer reviewed. Copyright 2018 Offshore Technology Conference. Reproduced by permission. This paper focuses on compressor systems associated with major production deferments. An advanced machine-learning approach is presented for determining anomalous behavior to predict a potential trip and probable root cause with sufficient warning to allow for intervention. This predictive-maintenance approach has the potential to reduce downtime associated with rotating-equipment failures. Introduction The first step in using a machine-learning system is to train the model to identify normal and abnormal operating conditions. The model can then classify real-time data from the equipment and indicate when the equipment’s performance strays outside the identified steady state. The ability to identify anomalies is a major difference between the proposed approach and traditional monitoring tools. With advances in digital technologies, correlations and warnings can be achieved in a matter of minutes, allowing engineers to take appropriate preventative action when they receive a failure warning. The authors used historical data for 2016 in their analysis of system efficiency in predicting failures. The proof-of-concept system correctly predicted 11 trip events over the course of the year, almost 50% of the 23 failures that occurred during that period. One of the more important findings was that the machine-learning model predicted many failures hours in advance. In one case, it gave 36 hours’ notice. The median period of notice for eight events that were subsequently analyzed was approximately 7 hours. Support Vector Machines (SVMs) SVMs are used in this study as a classifier for detecting abnormal machine states. SVMs were developed for binary classification. Some authors have argued that the SVM classifier has better results compared with techniques such as linear discriminant analysis and back-propagation neural networks. The compressor usually operates under normal working conditions. This poses a highly unbalanced problem for a two-class classification. Because of this, one-class classification using an SVM is implemented. The algorithm is trained on only normal data and creates a representation of this data. When the new points inferred are substantially different from the modeled class, they are labeled as outliers. Linear as well as radial kernel functions are explored. One of the properties of SVMs is that they may create a nonlinear decision boundary by projecting the data through nonlinear function to a space in higher dimension (Fig. 1). The one-class SVM creates a binary function that captures the region in the input space where most of the data exist. The function, thus formed, returns +1 for the region defined by training data points and –1 everywhere else.