Optimal Linear Classifier Research Articles

Investigations on interpretable nuclear-grade hydraulic damper monitoring and diagnosis from multi-source high-dimensional profile data

Condition monitoring and fault diagnosis is important to ensure smooth machine operation, in which the use of multi-source high-dimensional profile data composed of multiple physical signals collected by multiple sensors for machine health monitoring and fault diagnosis has not been fully explored. In this paper, interpretable monitoring and diagnostic methods are proposed to analyze multi-source high-dimensional profile data and identify equipment operational status. Firstly, polynomials of segmenting profiles as well as a Lasso model are established to obtain sparse features. Then, for offline data with historical fault labels, an optimal linear hyperplane classifier is constructed to interpretably study the impact of different features on a decision. For online data with only normal samples, constructing an optimal hypersphere enables the measurement of operational status. Large-scale nuclear-grade hydraulic damper profile data are used to demonstrate how the proposed interpretable models work and show their physical interpretation for monitoring and fault diagnosis.

Computational assessment of visual coding across mouse brain areas and behavioural states.

Our brain is bombarded by a diverse range of visual stimuli, which are converted into corresponding neuronal responses and processed throughout the visual system. The neural activity patterns that result from these external stimuli vary depending on the object or scene being observed, but they also change as a result of internal or behavioural states. This raises the question of to what extent it is possible to predict the presented visual stimuli from neural activity across behavioural states, and how this varies in different brain regions. To address this question, we assessed the computational capacity of decoders to extract visual information in awake behaving mice, by analysing publicly available standardised datasets from the Allen Brain Institute. We evaluated how natural movie frames can be distinguished based on the activity of units recorded in distinct brain regions and under different behavioural states. This analysis revealed the spectrum of visual information present in different brain regions in response to binary and multiclass classification tasks. Visual cortical areas showed highest classification accuracies, followed by thalamic and midbrain regions, with hippocampal regions showing close to chance accuracy. In addition, we found that behavioural variability led to a decrease in decoding accuracy, whereby large behavioural changes between train and test sessions reduced the classification performance of the decoders. A generalised linear model analysis suggested that this deterioration in classification might be due to an independent modulation of neural activity by stimulus and behaviour. Finally, we reconstructed the natural movie frames from optimal linear classifiers, and observed a strong similarity between reconstructed and actual movie frames. However, the similarity was significantly higher when the decoders were trained and tested on sessions with similar behavioural states. Our analysis provides a systematic assessment of visual coding in the mouse brain, and sheds light on the spectrum of visual information present across brain areas and behavioural states.

Does It Pay to Optimize AUC?

The Area Under the ROC Curve (AUC) is an important model metric for evaluating binary classifiers, and many algorithms have been proposed to optimize AUC approximately. It raises the question of whether the generally insignificant gains observed by previous studies are due to inherent limitations of the metric or the inadequate quality of optimization. To better understand the value of optimizing for AUC, we present an efficient algorithm, namely AUC-opt, to find the provably optimal AUC linear classifier in R2, which runs in O(n+n- log n+n-) where n+ and n- are the number of positive and negative samples respectively. Furthermore, it can be naturally extended to Rd in O(n+n-d-1 log (n+n-)) by recursively calling AUC-opt in lower-dimensional spaces. We prove the problem is NP-complete when d is not fixed, reducing from the open hemisphere problem. Compared with other methods, experiments show that AUC-opt achieves statistically significant improvements between 17 to 40 in R2 and 4 to 42 in R3 of 50 t-SNE training datasets. However, generally, the gain proves insignificant on most testing datasets compared to the best standard classifiers. Similar observations are found for nonlinear AUC methods under real-world datasets.

A novel multi-objective learning-to-rank method for software defect prediction

Search-Based Software Engineering (SBSE) is one of the techniques used for software defect prediction (SDP), in which search-based optimization algorithms are used to identify the optimal solution to construct a prediction model. As we know, the ranking methods of SBSE are used to solve insufficient sample problems, and the feature selection approaches of SBSE are employed to enhance the prediction model?s performance with curse-of-dimensionality or class imbalance problems. However, it is ignored that there may be a complex problem in the process of building prediction models consisting of the above problems. To address the complex problem, two multi-objective learning-to-rank methods are proposed, which are used to search for the optimal linear classifier model and reduce redundant and irrelevant features. To evaluate the performance of the proposed methods, excessive experiments have been conducted on 11 software programs selected from the NASA repository and AEEEM repository. Friedman?s rank test results show that the proposed method using NSGA-II outperforms other state-of-the-art singleobjective methods for software defect prediction.

An approach combining a new weight initialization method and constructive algorithm to configure a single Feedforward Neural Network for multi-class classification

This paper presents a new method for initializing weights in a Feedforward Neural Network (FNN) with a single hidden layer combined with a constructive approach to define the number of hidden units associated with the best classification performance. The strategy consists of defining an initial number of hidden units according to the classification problem, the linearization of the whole network around an equilibrium point and the determination of the initial weights and bias through the maximum approximation of the linearized model to the Optimal Linear Classifier (OLC) whose solution can be obtained analytically. The constructive algorithm comprises a gradual increase in the number of hidden units in such a way that at each training only the weights and bias associated with the new hidden units are initiated randomly while the weights and bias obtained from previous training are used as initial guesses. Additionally, the constructive algorithm seeks to ensure that the loss function of the trained networks decreases with the successive additions of hidden units. The proposed approach (Weight Initialization based on the Linearization of the Whole Neural Network combined with a new Constructive Algorithm, WILWNN-CA) is applied to synthetic and real datasets widely used as benchmark for multi-class classification problems. The comparison with conventional random weight initialization and other approaches involving different network topologies (and initialization strategies) shows that the proposed method is efficient and capable of providing success rates (correct classification rates) higher or similar to those achieved with existing methods.

Dynamic patterns of correlated activity in the prefrontal cortex encode information about social behavior.

New technologies make it possible to measure activity from many neurons simultaneously. One approach is to analyze simultaneously recorded neurons individually, then group together neurons which increase their activity during similar behaviors into an “ensemble.” However, this notion of an ensemble ignores the ability of neurons to act collectively and encode and transmit information in ways that are not reflected by their individual activity levels. We used microendoscopic GCaMP imaging to measure prefrontal activity while mice were either alone or engaged in social interaction. We developed an approach that combines a neural network classifier and surrogate (shuffled) datasets to characterize how neurons synergistically transmit information about social behavior. Notably, unlike optimal linear classifiers, a neural network classifier with a single linear hidden layer can discriminate network states which differ solely in patterns of coactivity, and not in the activity levels of individual neurons. Using this approach, we found that surrogate datasets which preserve behaviorally specific patterns of coactivity (correlations) outperform those which preserve behaviorally driven changes in activity levels but not correlated activity. Thus, social behavior elicits increases in correlated activity that are not explained simply by the activity levels of the underlying neurons, and prefrontal neurons act collectively to transmit information about socialization via these correlations. Notably, this ability of correlated activity to enhance the information transmitted by neuronal ensembles is diminished in mice lacking the autism-associated gene Shank3. These results show that synergy is an important concept for the coding of social behavior which can be disrupted in disease states, reveal a specific mechanism underlying this synergy (social behavior increases correlated activity within specific ensembles), and outline methods for studying how neurons within an ensemble can work together to encode information.

Structured discriminant analysis dictionary learning for pattern classification

Dictionary learning has been widely used in the field of pattern recognition. Although the existing synthesis dictionary learning methods have achieved impressive results, they need compute sparse codes using a time-consuming sparse coding procedure. As a dual viewpoint of synthesis dictionary learning, analysis dictionary learning (ADL) has attracted much attention due to its high efficiency and intuitive meaning in recent years. However, how to associate analysis dictionary atoms with class labels and learn a structured discriminant analysis dictionary, is still a challenging problem. In this paper, we propose a structured discriminant analysis dictionary learning (SDADL) method to learn a structured discriminant analysis dictionary that consists of the class-specific analysis sub-dictionaries associated with the corresponding classes. Specifically, we introduce a classification error term into SDADL model to learn an optimal linear classifier for classification. To obtain the discriminant analysis sparse codes, we introduce a discriminant analysis sparse code error term into SDADL model, which forces the samples from the same class to have similar analysis sparse codes. Moreover, we also introduce a structured discriminant term into SDADL model to improve the discrimination capability of both each class-specific analysis sub-dictionary and analysis sparse codes. An efficient iterative algorithm is also developed to solve the optimization problem of SDADL. In addition, we design a novel scheme for classification. Extensive experiments on five image datasets verify the effectiveness of SDADL for pattern classification.

Discriminative Dictionary Learning-Based Sparse Classification Framework for Data-Driven Machinery Fault Diagnosis

Data-driven machinery fault diagnosis is important for smart industrial systems to guarantee safety and reliability. However, the conventional data-driven fault diagnosis methods rely on the expert-designed features, which greatly affect the diagnosis performances. Inspired by the sparse representation-based classification (SRC) methods which can learn discriminative sparse features adaptively, we propose a novel discriminative dictionary learning based sparse classification (DDL-SC) framework for data-driven machinery fault diagnosis. The DDL-SC framework can jointly learn a discriminative dictionary for sparse representation and an optimal linear classifier for pattern recognition, which bridges the gaps between two separate processes, dictionary learning and classifier training in traditional SRC methods. In the learning stage, to enhance the discriminability of dictionary learning, we introduce the discriminative sparse code error along with the reconstruction error and classification error into the optimization objective. In the recognition stage, we employ sparse codes of testing signals with respect to the learned discriminative dictionary as inputs of the learned classifier, and promote the recognition performance by connecting a binary hard thresholding operator with the classifier predictions. The effectiveness of DDL-SC is evaluated on the planetary bearing fault dataset and gearbox fault dataset, indicating that DDL-SC yields the recognition accuracies of 99.73% and 99.41%, respectively. Besides, the comparative studies prove the superiority of DDL-SC over several state-of-the-art methods for data-driven machinery fault diagnosis.

Discriminative dictionary learning based sparse representation classification for intelligent fault identification of planet bearings in wind turbine

Planet bearing fault identification is an attractive but challenging task in wind turbine condition monitoring and fault diagnosis. Traditional fault characteristic frequency identification based diagnostic strategies are not sufficient for reliable planet bearing fault detection, due to complex physical configurations and modulation characteristics in planetary drivetrains. In this paper, we propose a discriminative dictionary learning based sparse representation classification (DDL-SRC) framework for intelligent planet bearing fault identification. Our framework could learn a reconstructive and discriminative dictionary for signal sparse representation and an optimal linear classifier for classification tasks simultaneously, which bridges the gap between dictionary learning and classifier training in traditional SRC methods. Specifically, the optimization objective introduces a novel regularization term called ‘discriminative sparse codes error’ and incorporates it with the reconstruction error and classification error. Thus, the dictionary learned by our framework possesses not only the reconstructive power for sparse representation but also the discriminative power for classifier training. The optimization formulation is efficiently solved using K-SVD and orthogonal matching pursuit algorithms. The experiment validations have been conducted for demonstrating the effectiveness and superiority of the proposed DDL-SRC framework over the state-of-the-art dictionary learning based SRC and deep convolutional neural network methods for intelligent planet bearing fault identification.

Reading-out task variables as a low-dimensional reconstruction of neural spike trains in single trials.

We propose a new model of the read-out of spike trains that exploits the multivariate structure of responses of neural ensembles. Assuming the point of view of a read-out neuron that receives synaptic inputs from a population of projecting neurons, synaptic inputs are weighted with a heterogeneous set of weights. We propose that synaptic weights reflect the role of each neuron within the population for the computational task that the network has to solve. In our case, the computational task is discrimination of binary classes of stimuli, and weights are such as to maximize the discrimination capacity of the network. We compute synaptic weights as the feature weights of an optimal linear classifier. Once weights have been learned, they weight spike trains and allow to compute the post-synaptic current that modulates the spiking probability of the read-out unit in real time. We apply the model on parallel spike trains from V1 and V4 areas in the behaving monkey macaca mulatta, while the animal is engaged in a visual discrimination task with binary classes of stimuli. The read-out of spike trains with our model allows to discriminate the two classes of stimuli, while population PSTH entirely fails to do so. Splitting neurons in two subpopulations according to the sign of the weight, we show that population signals of the two functional subnetworks are negatively correlated. Disentangling the superficial, the middle and the deep layer of the cortex, we show that in both V1 and V4, superficial layers are the most important in discriminating binary classes of stimuli.

Quantitative Imaging features Improve Discrimination of Malignancy in Pulmonary nodules

Pulmonary nodules are frequently detected radiological abnormalities in lung cancer screening. Nodules of the highest- and lowest-risk for cancer are often easily diagnosed by a trained radiologist there is still a high rate of indeterminate pulmonary nodules (IPN) of unknown risk. Here, we test the hypothesis that computer extracted quantitative features (“radiomics”) can provide improved risk-assessment in the diagnostic setting. Nodules were segmented in 3D and 219 quantitative features are extracted from these volumes. Using these features novel malignancy risk predictors are formed with various stratifications based on size, shape and texture feature categories. We used images and data from the National Lung Screening Trial (NLST), curated a subset of 479 participants (244 for training and 235 for testing) that included incident lung cancers and nodule-positive controls. After removing redundant and non-reproducible features, optimal linear classifiers with area under the receiver operator characteristics (AUROC) curves were used with an exhaustive search approach to find a discriminant set of image features, which were validated in an independent test dataset. We identified several strong predictive models, using size and shape features the highest AUROC was 0.80. Using non-size based features the highest AUROC was 0.85. Combining features from all the categories, the highest AUROC were 0.83.

A dynamic linear model for heteroscedastic LDA under class imbalance

Linear Discriminant Analysis (LDA) yields the optimal Bayes classifier for binary classification for normally distributed classes with equal covariance. To improve the performance of LDA, heteroscedastic LDA (HLDA) that removes the equal covariance assumption has been developed. In this paper, we show using first and second-order optimality conditions that the existing approaches either have no principled computational procedure for optimal parameter selection, or underperform in terms of the accuracy of classification and the area under the receiver operating characteristics curve (AUC) under class imbalance. Using the same optimality conditions, we then derive a dynamic Bayes optimal linear classifier for heteroscedastic LDA that is optimised via an efficient iterative procedure, which is robust against class imbalance. Experimental work is conducted on two artificial and eight real-world datasets. Our results show that the proposed algorithm compares favourably with the existing heteroscedastic LDA procedures as well as the linear support vector machine (SVM) in terms of the error rate, but is superior to all the algorithms in terms of the AUC under class imbalance. The fast training time of the proposed algorithm also encourages its use for large-data applications that show high incidence of class imbalance, such as in human activity recognition.

Linear classifier design in the weight space

In this paper, we propose a linear classifier design method in the weight space. A linear classifier is completely determined by a weight vector. To design a linear classifier is equivalent to finding a weight vector. When there are a number of training samples, each training sample represents a plane in the weight space. On one side of the plane, the training sample is correctly classified while it is incorrectly classified on the other side. Thus, finding the optimal linear classifier can be formulated as finding a subspace that provides the best classification accuracy. In this paper, we have developed an algorithm to find such optimal subspaces in the weight space. Although we could not find the optimal linear classifier due to prohibitive computational complexity, the proposed design method produced noticeable improvements in some cases. Experimental results show that the proposed linear classifier performed better than or equivalent to existing linear classifiers.

Joint Discriminative Dictionary and Classifier Learning for ALS Point Cloud Classification

To efficiently recognize on-ground objects in airborne laser scanning (ALS) point clouds, we design a method that jointly learns a discriminative dictionary and a classifier. In the method, the point cloud is segmented into hierarchical point clusters, which are organized by a tree structure. Then, the feature of each point cluster is extracted. The feature of a leaf node is obtained by aggregating the features of all its parent nodes. The feature of the leaf node is called the hierarchical aggregation feature. The hierarchical aggregation features are encoded by sparse coding. We introduce a new label consistency constraint called “discriminative sparse-code error,” and combine it with the reconstruction error, the classification error, and $L_{1}$ -norm sparsity constraint to form a unified objective function. The objective function is efficiently solved by using the proposed label consistency feature sign method. We obtain an overcomplete discriminative dictionary and an optimal linear classifier. Experiments performed on different ALS point cloud scenes have shown that the hierarchical aggregation features combined with the learned classifier can significantly enhance the classification results, and also demonstrated the superior performance of our method over other techniques in point cloud classification.

Effective micro-expression recognition using relaxed K-SVD algorithm

Regarded as its subtlety, micro-expression is a challenging research problem. In the paper, we propose a relax K-SVD classifier (RK-SVD) for micro-expression recognition. RK-SVD minimizes the variance of sparse coefficients to address the similarity of same classes and the distinctiveness of different classes in sparse coefficients. In addition, reconstruction error and classification error are also considered. The optimization is implemented by the K-SVD algorithm and stochastic gradient descent algorithm. Finally a single overcomplete dictionary and an optimal linear classifier are learned simultaneously. We show that RK-SVD can effectively recognize micro-expression under three spontaneous micro-expression datasets including SMIC, CASME, and CASME II.

Dictionary learning based on discriminative energy contribution for image classification

This paper combines the discriminative feature extraction and effective classifier construction into a single framework to learn a structured discriminative dictionary for image classification. Due to the fact that the discriminative signal lie in a low dimensional subspace and can be well represented only via a few atoms of the learned dictionary, this paper addresses the feature extraction via learning a dictionary, whose sub dictionaries preserve correspondence to the class labels, and an optimal linear classifier jointly based on the structure of energy contribution. Based on the discriminative energy contributions, we are searching the discriminative feature for classification rather than reconstructing the data accurately. In addition, with the assumption that the classifier has a specific property which is similar with the dictionary, we learn a classifier to make the dictionary optimal and have a low cost on classifying. Experiment results on the several databases to specific classification tasks are conducted to verify the efficacy of the proposed method compared with the state-of-the-art dictionary learning for classification methods.

Cross-domain action recognition via collective matrix factorization with graph Laplacian regularization

This paper investigates the problem of cross-domain action recognition. Specifically, we present a cross-domain action recognition framework by utilizing some labeled data from other data sets as the auxiliary source domain. It is a challenging task as data from different domains may have different feature distribution. To map data from different domains into the same abstract space and boost the action recognition performance, we propose a method named collective matrix factorization with graph Laplacian regularization (CMFGLR). Our approach is built upon the technique of collective matrix factorization, which simultaneously learns a common latent space, linear projection matrices for obtaining semantic representations, and an optimal linear classifier. Moreover, we explore the label consistency across different domain and the local geometric consistency in each domain and obtain a graph Laplacian regularization term to enhance the discrimination of learned features. Experimental results verify that CMFGLR significantly outperforms several state-of-the-art methods.

Discriminative dictionary learning via Fisher discrimination K-SVD algorithm

An efficient and discriminative dictionary structure is important for pattern classification. Considering the discrimination of the sparse coding coefficients, a method named Fisher Discriminative K-SVD (FD-KSVD) is proposed to learn an over-complete discriminative dictionary and an optimal linear classifier. In FD-KSVD, the Fisher discrimination criterion is imposed on the sparse coding coefficients to make them discriminative through small within-class scatter and big between-class scatter. Hence not only the reconstruction error can be used, but also the discriminative error and the classification error have been represented. The optimization is employed by the Iterative Projective Method and K-SVD method alternatively. Experimental results demonstrate that our proposed method outperforms many other state-of-the art methods for face and palmprint recognition.

Selection of Support Vector Machines based classifiers for credit risk domain

This paper describes an approach for credit risk evaluation based on linear Support Vector Machines classifiers, combined with external evaluation and sliding window testing, with focus on application on larger datasets. It presents a technique for optimal linear SVM classifier selection based on particle swarm optimization technique, providing significant amount of focus on imbalanced learning issue. It is compared to other classifiers in terms of accuracy and identification of each class. Experimental classification performance results, obtained using real world financial dataset from SEC EDGAR database, lead to conclusion that proposed technique is capable to produce results, comparable to other classifiers, such as logistic regression and RBF network, and thus be can be an appealing option for future development of real credit risk evaluation models.

Online Learning a High-Quality Dictionary and Classifier Jointly for Multitask Object Tracking

This article formulates object tracking in a particle filter framework as a binary classification problem. The method effectively exploits a priori information from training data to learn online a compact and discriminative dictionary. The method incorporates the class label information into the dictionary learning process as the classification error term and idea coding regularization term, respectively. Combined with the traditional reconstruction error, a total objective function for dictionary learning is constructed. By minimizing the total object function, the approach jointly obtains a high-quality dictionary and optimal linear classifier. Combined with multitask sparse coding, the best candidate is selected by jointly evaluating the reconstructive error and classification error. As the tracking continues, the proposed algorithm alternates between multitask sparse coding and dictionary updating. Experimental evaluations on challenging video sequences show that the proposed algorithm performs favorably against state-of-the-art methods in terms of effectiveness, accuracy, and robustness.