Discriminative Objective Function Research Articles

Twin SVM for conditional probability estimation in binary and multiclass classification

In this paper, we estimate the conditional probability function by presenting a new twin SVM model (CPTWSVM) in binary and multiclass classification problems. The motivation of CPTWSVM is to implement the empirical risk minimization on training data, which is hard to realize in traditional twin SVMs. In each subproblem of CPTWSVM, it measures the empirical risk and outputs the corresponding probability estimate of each class, which eliminates the problems of inconsistent measurement in twin SVMs. Though an additional discriminant objective function is introduced, the optimization problem size of each subproblem is smaller than conditional probability SVM, and is solved by block decomposition algorithm efficiently. In addition, we extend CPTWSVM to multiclass classification by estimating the conditional probability of each class, and maintaining the above properties. Numerical experiments on benchmark and real application datasets demonstrate that CPTWSVM outputs the estimate of probability and the data projection well, resulting in better generalization ability than some leading TWSVMs communities, in terms of binary and multiclass classification.

Learning latent local manifold embeddings for hyperspectral feature extraction

Hyperspectral images with an immense number of spectral bands provide abundant discriminant information for accurate land-cover classification in the remote sensing field. However, these narrow and adjacent bands contain a large amount of redundant information. Analyzing these images always requires a huge storage space with expensive computational costs. Furthermore, their high correlation coefficient would lead to the Hughes phenomenon, hindering the improvement of classification performance. We propose a linear semi-supervised hyperspectral feature extraction method L3ME to learn latent local manifold embeddings. Although labeled samples are beneficial to construct learning models, their number is always limited in real-world tasks. The motivation of this paper is to jointly enhance the contributions of labeled and unlabeled samples for learning local manifold structures of hyperspectral images. Features in labeled samples are extracted by two procedures, the adaptive patch alignment framework and integrated intraclass-interclass relationships, from different perspectives. The former aims to solve the problem of the uneven distribution of classes by introducing spectral angle based adaptive parameters. The latter aims to solve the problem of the uneven distribution of samples by constructing several adjacency graphs. The locality preserving projection is capable of preserving the local neighborhood structure of samples. A penalty for sparse regularization is cleverly integrated into the proposed linear discriminant objective function, which is optimized using a novel updating strategy. The convergence of L3ME is proved in detail and analyzed in this paper. Experiments on three typical hyperspectral datasets illustrate the effectiveness of the proposed method over some state-of-the-art techniques. The implementation of L3ME is available at https://github.com/biowby/L3ME.

Attention gated tensor neural network architectures for speech emotion recognition

In an attempt to make Human-Computer Interactions more natural, we propose the use of Tensor Factorized Neural Networks (TFNN) and Attention Gated Tensor Factorized Neural Network (AG-TFNN) for Speech Emotion Recognition (SER) task. Standard speech representations such as 2D and 3D Mel-Spectrogram and Temporal Modulation Spectrogram is explored to investigate the emotion salient information capturing effectiveness of the Tensor Factorization based architectures. The hidden layers are explained as Deep Tensor Factorization based on the Tucker Decomposition but with a unified discriminative objective function to learn the factor matrices in a discriminative sense. The core tensor produced in each hidden layer is the feature associated with that factorisation layer. Mel Spectrograms are naturally in 2D tensor form, and thus TFNN and AG-TFNN becomes an appropriate choice over baselines such as Convolutional Neural Network (CNN) and Long Short-Term Memory (LSTM) by providing reduced parameters to be learned and a simple architecture. Experiments conducted on standard emotional speech datasets- Emo-DB and IEMOCAP shows that TFNN and AG-TFNN surpasses the state-of-the-art given by CNN+LSTM combination with fewer number of parameters.

Semisupervised Graph Convolution Deep Belief Network for Fault Diagnosis of Electormechanical System With Limited Labeled Data

The labeled monitoring data collected from the electromechanical system is limited in the real industries; traditional intelligent fault diagnosis methods cannot achieve satisfactory accurate diagnosis results. To deal with this problem, an intelligent fault diagnosis method for electromechanical system based on a new semisupervised graph convolution deep belief network algorithm is proposed in this article. Specifically, the labeled and unlabeled samples are first employed to design a new adaptive local graph learning method for constructing the graph neighbor relationship. Meanwhile, the labeled samples are applied to describe the discriminative structure information of data via the latest circle loss. Finally, the local and discriminative objective functions are reconstructed under the semisupervised learning framework. The experimental results from the motor-bearing system demonstrate that the method can achieve 98.66 % accuracy with only 10 % of training labeled data, which indicates that it is a promising semisupervised intelligent fault diagnosis method.

Deep Clustering: On the Link Between Discriminative Models and K-Means.

In the context of recent deep clustering studies, discriminative models dominate the literature and report the most competitive performances. These models learn a deep discriminative neural network classifier in which the labels are latent. Typically, they use multinomial logistic regression posteriors and parameter regularization, as is very common in supervised learning. It is generally acknowledged that discriminative objective functions (e.g., those based on the mutual information or the KL divergence) are more flexible than generative approaches (e.g., K-means) in the sense that they make fewer assumptions about the data distributions and, typically, yield much better unsupervised deep learning results. On the surface, several recent discriminative models may seem unrelated to K-means. This study shows that these models are, in fact, equivalent to K-means under mild conditions and common posterior models and parameter regularization. We prove that, for the commonly used logistic regression posteriors, maximizing the L2 regularized mutual information via an approximate alternating direction method (ADM) is equivalent to minimizing a soft and regularized K-means loss. Our theoretical analysis not only connects directly several recent state-of-the-art discriminative models to K-means, but also leads to a new soft and regularized deep K-means algorithm, which yields competitive performance on several image clustering benchmarks.

Recurrent neural network language generation for spoken dialogue systems

Natural Language Generation (NLG) is a critical component of spoken dialogue systems and it has a significant impact both on usability and perceived quality. Most existing NLG approaches in common use employ rules and heuristics and tend to generate rigid and stylised responses without the natural variation of human language. Moreover, these limitations also add significantly to development costs and make the delivery of cross-domain, cross-lingual dialogue systems especially complex and expensive. The first contribution of this paper is to present RNNLG, a Recurrent Neural Network (RNN)-based statistical natural language generator that can learn to generate utterances directly from dialogue act – utterance pairs without any predefined syntaxes or semantic alignments. The presentation includes a systematic comparison of the principal RNN-based NLG models available. The second contribution, is to test the scalability of the proposed system by adapting models from one domain to another. We show that by pairing RNN-based NLG models with a proposed data counterfeiting method and a discriminative objective function, a pre-trained model can be quickly adapted to different domains with only a few examples. All of the findings presented are supported by both corpus-based and human evaluations.

Internal Emotion Classification Using EEG Signal With Sparse Discriminative Ensemble

Among various physiological signal acquisition methods for the study of the human brain, EEG (Electroencephalography) is more effective. EEG provides a convenient, non-intrusive, and accurate way of capturing brain signals in multiple channels at fine temporal resolution. We propose an ensemble learning algorithm for automatically computing the most discriminative subset of EEG channels for internal emotion recognition. Our method describes an EEG channel using kernel-based representations computed from the training EEG recordings. For ensemble learning, we formulate a graph embedding linear discriminant objective function using the kernel representations. The objective function is efficiently solved via sparse non-negative principal component analysis and the final classifier is learned using the sparse projection coefficients. Our algorithm is useful in reducing the amount of data while improving computational efficiency and classification accuracy at the same time. The experiments on publicly available EEG dataset demonstrate the superiority of the proposed algorithm over the compared methods.

Learning to rank with relational graph and pointwise constraint for cross-modal retrieval

Cross-modal retrieval (i.e., image–query–text or text–query–image) is a hot research topic for multimedia information retrieval, but the heterogeneity gap between different modalities generates a critical challenge for multimodal data. Some researchers regard the cross-modal retrieval as a leaning to rank task, and they usually consider to measure similarity between two different modalities in the embedding shared subspace. However, previous methods almost pay more attention to construct a discriminative objective function to optimize common space, ignoring to exploit correlation between the single modality. In this paper, we consider the cross-modal retrieval task, from the perspective of optimizing ranking model, as a listwise ranking problem, and propose a novel method called learning to rank with relational graph and pointwise constraint ( $$ {\text{LR}}^{2} {\text{GP}} $$ ). In $$ {\text{LR}}^{2} {\text{GP}} $$ , we first propose a discriminative ranking model, which makes use of the relation between the single modality to improve ranking performance so as to learn an optimal embedding common subspace. Then, a pointwise constraint is introduced in the low-dimension embedding subspace to make up for the real loss in the training phase since listwise method introduced merely considers directly optimize latent permutation from the perspective of the overall. Finally, a dynamic interpolation algorithm, which gradually transits from pointwise and pairwise to listwise learning, is selected to deal with the problem of fusion of loss function reasonable. Experiments on the benchmark datasets about Wikipedia and Pascal demonstrate the effectiveness for proposed method.

X-ray CT image analysis for morphology of muskmelon seed in relation to germination

Internal morphological damage can have critical effects on the development and germination power of seeds. This study investigates the morphological characteristics of naturally aged muskmelon seed in relation to germination ability. An X-ray microCT scanner was employed to generate CT images and then several image processing techniques such as re-slicing, contrast enhancement, noise reduction, and segmentation were performed on the images. Afterwards, fifteen preprocessed images were nominated from each sample, and features of interest (i.e., local binary pattern, Gabor, local Fourier (FFT), texture, contrast and Haralick textural (Tx) features) were extracted. The sequential forward selection (SFS) method was applied as a search strategy to identify the most relevant features using a variety of different objective functions. It was determined that the Fisher discriminant objective function performed the best. A germination test was performed to evaluate the seed viability and the information was used to construct the training and validation data set. The seeds were divided into 2 groups: viable (group-1) and non-viable (group-0). Different classifiers were probed to determine the optimal performer, where the linear discriminant classifier resulted in an accuracy of 98.9%, with 10-fold cross-validation using eighteen selected features. The findings of this study indicate that CT imaging is a potential tool for the classification of seeds based on the characterisation of internal morphologically.

When Deep Learning Meets Metric Learning: Remote Sensing Image Scene Classification via Learning Discriminative CNNs

Remote sensing image scene classification is an active and challenging task driven by many applications. More recently, with the advances of deep learning models especially convolutional neural networks (CNNs), the performance of remote sensing image scene classification has been significantly improved due to the powerful feature representations learnt through CNNs. Although great success has been obtained so far, the problems of within-class diversity and between-class similarity are still two big challenges. To address these problems, in this paper, we propose a simple but effective method to learn discriminative CNNs (D-CNNs) to boost the performance of remote sensing image scene classification. Different from the traditional CNN models that minimize only the cross entropy loss, our proposed D-CNN models are trained by optimizing a new discriminative objective function. To this end, apart from minimizing the classification error, we also explicitly impose a metric learning regularization term on the CNN features. The metric learning regularization enforces the D-CNN models to be more discriminative so that, in the new D-CNN feature spaces, the images from the same scene class are mapped closely to each other and the images of different classes are mapped as farther apart as possible. In the experiments, we comprehensively evaluate the proposed method on three publicly available benchmark data sets using three off-the-shelf CNN models. Experimental results demonstrate that our proposed D-CNN methods outperform the existing baseline methods and achieve state-of-the-art results on all three data sets.

An alternate method between generative objective and discriminative objective in training classification Restricted Boltzmann Machine

As a derivative of Restricted Boltzmann Machine (RBM), classification RBM (ClassRBM) has been an effective classifier. However, there are still many disadvantages in training ClassRBM. For example, the prediction accuracy with the generative objective function (GenF) is not high, and the training process with the discriminative objective function (DisF) and the hybrid RBM (HDRBM) are time-consuming. In this paper, we propose an alternate method between Generative Objective and Discriminative Objective (ANGD) to train ClassRBM after examining the training process of GenF and DisF. At each iteration step of ANGD, the parameters of ClassRBM are firstly updated by maximizing GenF when the training accuracy can be improved, then modified by maximizing DisF. This process is repeated until some stop criterion is met. ANGD achieves a good prediction accuracy with a relatively less training cost because it utilizes the complementation of GenF and DisF. The comparative experiments on five datasets show that ANGD beats GenF, DisF and HDRBM. As a whole, the accuracy of ANGD is the best and the stability is acceptable, and the training cost of ANGD is also the best on the datasets with a large size. The training efficiency of ANGD is the best among the four methods.

Pedestrian search in surveillance videos by learning discriminative deep features

Abstract Searching for a target person in videos captured by many non-overlapped cameras is an important yet challenging problem in the fields of intelligent video surveillance. Person re-identification is a key technique in the person searching task. In this paper, we propose a discriminative objective function to learn deep CNN features for person re-id. Specially, the proposed objective function reduces distances between instances belonging to the same person, and enlarges distances between instances belonging to different persons at the same time. With the goal of inter-class dispersion and intra-class compactness, the obtained deep features can be more discriminative than many traditional training objectives, e.g. softmax, contrastive and triplet objective functions. Extensive experiments on person re-id benchmarks validated the superiority of the proposed objective function. Based on the proposed person re-identification algorithm, we implement a pedestrian search system by integrating three components: pedestrian detection, multi-person tracking and person re-identification together. The whole system is evaluated on a Cross-Camera Pedestrian Search Challenge and achieves superior performances on the evaluation set.

Postharvest assessment of undesirable fibrous tissue (choking hazard) in fresh processing carrots using Vis/NIR hyperspectral images

This research was designed to develop and test an automatic image analysis algorithm to detect the presence (pre-dicing) of undesirable fibrous carrot (Daucus carota L.) tissue. Fibrous carrot dices are difficult to detect, and are highly problematic when found in ready-to-eat infant food, where they might represent a choking hazard (safety concern). A Visible/Near-InfraRed (Vis/NIR) hyperspectral imaging (321-1088 nm) system was used to obtain a set of 520-images per sample, from 1233 sections (samples). Carrots were collected during the 2013 and 2014 harvesting seasons. Classification accuracy per sample was evaluated by comparing the classes obtained using Vis/NIR hyperspectral images against their undesirable fibrous tissue class, based on the industry-simulated invasive quality assessment (% of fiber). Class-0 represents fibrous-free samples, and class-1 denotes samples containing fibrous tissue. After Vis/NIR image preprocessing, cropping, selection, and segmentation, 3135 grayscale intensity and textural features were extracted per sample from 15 selected Vis/NIR images. A 4-fold cross-validation Neural-Network-Classifier with a performance accuracy of 86.4±2.1% was developed using 140 relevant features, which were selected using a sequential forward selection algorithm with the Fisher discriminant objective function. Findings showed that this methodology is an objective, accurate, and reliable tool to determine the presence of undesirable fibrous tissue in processing carrots, and would be applicable to an automated noninvasive inline sorting system.

Postharvest noninvasive classification of tough-fibrous asparagus using computed tomography images

This research was devised to evaluate Computed Tomography (CT) for asparagus fibrousness detection and more specifically develop and test an automatic image analysis method (algorithm) to classify CT images obtained from 859 asparagus (Asparagus officinalis L.) segment (samples), collected during two harvesting seasons (2014 and 2015). Classification accuracy was calculated by comparing the classes obtained using a combination of imaging, image processing, feature extraction, and classification schemes per asparagus segment against an industry-simulated invasive quality assessment.Grayscale intensity and textural features, 3762 total, were extracted from minimum and maximum resultant images from three CT planer views. A 4-fold cross-validation linear discriminant classifier with a performance accuracy of 91.2% was developed using 75 relevant features, which were selected using a sequential forward selection algorithm with the Fisher discriminant objective function. This objective method is accurate in determining the presence of tough-fibrous tissue in asparagus, which demonstrates a potential for such technology to objectively forecast asparagus quality and thus supports the asparagus industry through optimizing consumer acceptability and product utilization.

Efficient and precise interactive hand tracking through joint, continuous optimization of pose and correspondences

Fully articulated hand tracking promises to enable fundamentally new interactions with virtual and augmented worlds, but the limited accuracy and efficiency of current systems has prevented widespread adoption. Today's dominant paradigm uses machine learning for initialization and recovery followed by iterative model-fitting optimization to achieve a detailed pose fit. We follow this paradigm, but make several changes to the model-fitting, namely using: (1) a more discriminative objective function; (2) a smooth-surface model that provides gradients for non-linear optimization; and (3) joint optimization over both the model pose and the correspondences between observed data points and the model surface. While each of these changes may actually increase the cost per fitting iteration, we find a compensating decrease in the number of iterations. Further, the wide basin of convergence means that fewer starting points are needed for successful model fitting. Our system runs in real-time on CPU only, which frees up the commonly over-burdened GPU for experience designers. The hand tracker is efficient enough to run on low-power devices such as tablets. We can track up to several meters from the camera to provide a large working volume for interaction, even using the noisy data from current-generation depth cameras. Quantitative assessments on standard datasets show that the new approach exceeds the state of the art in accuracy. Qualitative results take the form of live recordings of a range of interactive experiences enabled by this new approach.

Postharvest noninvasive assessment of undesirable fibrous tissue in fresh processing carrots using computer tomography images

This research was designed to develop and test an automatic image analysis method (algorithm) to classify CT images obtained from 1233 carrot (Daucus carota L.) sections (samples), collected during the 2013 and 2014 harvesting seasons. Classification accuracy was evaluated by comparing the classes obtained using eighteen CT images per carrot section to their undesirable fibrous tissue class, based on the industry-simulated invasive quality assessment (% of fiber). Class-0 represents fibrous-free samples, and class-1 denotes samples containing fibrous tissue.After CT image preprocessing, cropping, and segmentation, 3762 grayscale intensity and textural features were extracted from the eighteen CT images per sample. A 4-fold cross-validation linear discriminant classifier with a performance accuracy of 87.9% was developed using 95 relevant features, which were selected using a sequential forward selection algorithm with the Fisher discriminant objective function. This objective method is accurate in determining the presence of undesirable fibrous tissue in pre-processed carrots.

Identification of Predictive Cis-Regulatory Elements Using a Discriminative Objective Function and a Dynamic Search Space.

The generation of genomic binding or accessibility data from massively parallel sequencing technologies such as ChIP-seq and DNase-seq continues to accelerate. Yet state-of-the-art computational approaches for the identification of DNA binding motifs often yield motifs of weak predictive power. Here we present a novel computational algorithm called MotifSpec, designed to find predictive motifs, in contrast to over-represented sequence elements. The key distinguishing feature of this algorithm is that it uses a dynamic search space and a learned threshold to find discriminative motifs in combination with the modeling of motifs using a full PWM (position weight matrix) rather than k-mer words or regular expressions. We demonstrate that our approach finds motifs corresponding to known binding specificities in several mammalian ChIP-seq datasets, and that our PWMs classify the ChIP-seq signals with accuracy comparable to, or marginally better than motifs from the best existing algorithms. In other datasets, our algorithm identifies novel motifs where other methods fail. Finally, we apply this algorithm to detect motifs from expression datasets in C. elegans using a dynamic expression similarity metric rather than fixed expression clusters, and find novel predictive motifs.

Adaptation of motor imagery EEG classification model based on tensor decomposition

Objective. Session-to-session nonstationarity is inherent in brain–computer interfaces based on electroencephalography. The objective of this paper is to quantify the mismatch between the training model and test data caused by nonstationarity and to adapt the model towards minimizing the mismatch. Approach. We employ a tensor model to estimate the mismatch in a semi-supervised manner, and the estimate is regularized in the discriminative objective function. Main results. The performance of the proposed adaptation method was evaluated on a dataset recorded from 16 subjects performing motor imagery tasks on different days. The classification results validated the advantage of the proposed method in comparison with other regularization-based or spatial filter adaptation approaches. Experimental results also showed that there is a significant correlation between the quantified mismatch and the classification accuracy. Significance. The proposed method approached the nonstationarity issue from the perspective of data-model mismatch, which is more direct than data variation measurement. The results also demonstrated that the proposed method is effective in enhancing the performance of the feature extraction model.

Discriminative learning of generative models: large margin multinomial mixture models for document classification

In this paper, a novel discriminative learning method is proposed to estimate generative models for multi-class pattern classification tasks, where a discriminative objective function is formulated with separation margins according to certain discriminative learning criterion, such as large margin estimation (LME). Furthermore, the so-called approximation-maximization (AM) method is proposed to optimize the discriminative objective function w.r.t. parameters of generative models. The AM approach provides a good framework to deal with latent variables in generative models and it is flexible enough to discriminatively learn many rather complicated generative models. In this paper, we are interested in a group of generative models derived from multinomial distributions. Under some minor relaxation conditions, it is shown that the AM-based discriminative learning methods for these generative models result in linear programming (LP) problems that can be solved effectively and efficiently even for rather large-scale models. As a case study, we have studied to learn multinomial mixture models (MMMs) for text document classification based on the large margin criterion. The proposed methods have been evaluated on a standard RCV1 text corpus. Experimental results show that large margin MMMs significantly outperform the conventional MMMs as well as pure discriminative models such as support vector machines (SVM), where over 25 % relative classification error reduction is observed in three independent RCV1 test sets.

Postharvest noninvasive assessment of fresh chestnut (Castanea spp.) internal decay using computer tomography images

X-ray computed tomography (CT) is an effective noninvasive tool to visualize fresh agricultural commodities’ internal components and quality attributes, including those of chestnuts (Castanea spp). There is no procedure to automatically, effectively and efficiently classify fresh commodities from a continuous inline flow through a CT system. If the information obtained by CT scanning of fresh agricultural commodities is to be used in an industrial application (e.g. inline sorting), automated interpretation of CT images is essential. For this purpose, an image analysis method (algorithm) for the automatic classification of CT images obtained from 2848 fresh chestnuts (cv. ‘Colossal’ and ‘Chinese seedlings’), during the harvesting years from 2009 to 2012, was developed and tested. Classification accuracy was evaluated by comparing the classes obtained from six CT images per chestnut to their internal quality assessment. An experienced human rater performed internal quality assessment by visually and invasively rating fresh chestnut internal decay severity (quality) into 5-, 3- and 2-classes.After CT image preprocessing, cropping and segmentation, 1194 grayscale intensity and textural features were extracted from six resultant CT images per sample. Relevant features were selected using a sequential forward selection algorithm with the Fisher discriminant objective function. 86, 155 and 126 features were effective in designing a quadratic discriminant classifier with a 4-fold cross-validation with a performance accuracy of 85.9%, 91.2% and 96.1% for 5, 3 and 2 classes, respectively. This method is accurate and objective in determining fresh chestnut internal quality, and the methodology is applicable to automatic noninvasive inline CT sorting system development.