Sparse Linear Discriminant Analysis Research Articles

High dimensional discriminant analysis under weak sparsity

. In this work, we consider the discriminant analysis under the weak sparsity where many entries of the parameters are nearly zero. We develop a unified LASSO-typed framework to estimate the parameters for sparse discriminant analysis and derive the general non asymptotic error bound under the weak sparsity condition. As applications, we revisit the sparse linear discriminant analysis and sparse quadratic discriminant analysis. We establish the consistency of the estimators and also the misclassification error rate. These results extend the sparse discriminant analysis methods to weak sparsity setting and refine the existing theoretical results.

Towards robust and sparse linear discriminant analysis for image classification

Linear discriminant analysis (LDA) is a popular dimensionality reduction technique that has been widely used in pattern recognition. However, there exist a large number of redundant features and corrupted noise in real-world applications, which makes the performance of existing LDA methods degrade and thus leads to a decrease in classification accuracy. To address the above issues, we propose a novel robust and sparse LDA formulation dubbed RSLDA+. The key idea is introducing the mixed sparse regularization, i.e., ℓ0-norm plus ℓ2,0-norm, for feature representation and enforce ℓ0-norm for noise reduction. Furthermore, an optimization algorithm based on the alternating direction method of multipliers (ADMM) is developed in combination with hard thresholding operators. Extensive experiments on six common image datasets verify that the proposed RSLDA+ outperforms state-of-the-art LDA variants in classification accuracy. In addition, the ablation, robustness, convergence, stability, and sparsity are analyzed in detail. The results suggest that the proposed RSLDA+ provides an effective and robust method for image classification.

Model-based impending lithium battery terminal voltage collapse detection via data-driven and machine learning approaches

Lithium battery is an important power source of an electrical vehicle (EV). Practically, when a battery is about to fail, it needs to be removed from the load because keeping it connected can lead to permanent damage. Hence, it is important to detect battery failure to sustain the lifespan of the battery and avoid safety issues which will contribute to the sustainability of the EV. However, determining the effective disconnecting or discharging moment of the battery remains a challenging issue in EV applications. In order to solve the abovementioned problem, a control framework and maximum likelihood estimation are proposed to estimate the parameters of a battery model. In addition, both the sparse representation and dynamic mode decomposition (DMD) approaches are applied to protect the physical battery unit from a failure scenario. For comparison purposes, the proposed framework is compared to a Lyapunov-based detection approach along with neural network (NN) and linear discriminant analysis (LDA). Moreover, several state estimation algorithms are applied to estimate battery state of charge (SOC) at each time step: extended Kalman filter, extended Kalman smooth variable structure filter, and cubature Kalman filter. Finally, the experimental results showed that the proposed DMD approach outperforms the sparse representation, NN, LDA, and Lyapunov-based approaches in terms of detection accuracy. Moreover, the proposed DMD strategy is more robust towards the inaccuracy of the estimated SOC. Besides that, the proposed battery model parameter estimation approach exhibited fast processing time; therefore, it is a reliable choice for recomputing the model parameters from time to time to compensate aging effect. An autonomous unmanned aerial vehicle (UAV) was simulated as a proof-of-concept in a MATLAB environment to verify the detection performance of the proposed methods. Both actual and UAV results showed that the DMD demonstrated more robust detection performance than other methods in terms of processing time and detection accuracy.

Improving the second-tier classification of methylmalonic acidemia patients using a machinelearning ensemble method.

Methylmalonic acidemia (MMA) is a disorder of autosomal recessive inheritance, with an estimated prevalence of 1:50,000. First-tier clinical diagnostic tests often return many false positives [five false positive (FP): one true positive (TP)]. In this work, our goal was to refine a classification model that can minimize the number of false positives, currently an unmet need in the upstream diagnostics of MMA. We developed machinelearning multivariable screening models for MMA with utility as a secondary-tier tool for false positives reduction. We utilized mass spectrometry-based features consisting of 11 amino acids and 31 carnitines derived from dried blood samples of neonatal patients, followed by additional ratio feature construction. Feature selection strategies (selection by filter, recursive feature elimination, and learned vector quantization) were used to determine the input set for evaluating the performance of 14 classification models to identify a candidate model set for an ensemble model development. Our work identified computational models that explore metabolic analytes to reduce the number of false positives without compromising sensitivity. The best results [area under the receiver operating characteristic curve (AUROC) of 97%, sensitivity of 92%, and specificity of 95%] were obtained utilizing an ensemble of the algorithms random forest, C5.0, sparse linear discriminant analysis, and autoencoder deep neural network stacked with the algorithm stochastic gradient boosting as the supervisor. The model achieved a good performance trade-off for a screening application with 6% false-positive rate (FPR) at 95% sensitivity, 35% FPR at 99% sensitivity, and 39% FPR at 100% sensitivity. The classification results and approach of this research can be utilized by clinicians globally, to improve the overall discovery of MMA in pediatric patients. The improved method, when adjusted to 100% precision, can be used to further inform the diagnostic process journey of MMA and help reduce the burden for patients and their families.

Sr-LDA:Sparse and Reduced-Rank Linear Discriminant Analysis for High Dimensional Matrix

Sr-LDA:Sparse and Reduced-Rank Linear Discriminant Analysis for High Dimensional Matrix

Sparse Linear Discriminant Analysis Based on lq Regularization

Linear discriminant analysis plays an important role in feature extraction, data dimensionality reduction, and classification. With the progress of science and technology, the data that need to be processed are becoming increasingly large. However, in high-dimensional situations, linear discriminant analysis faces two problems: the lack of interpretability of the projected data since they all involve all p features, which are linear combinations of all features, as well as the singularity problem of the within-class covariance matrix. There are three different arguments for linear discriminant analysis: multivariate Gaussian model, Fisher discrimination problem, and optimal scoring problem. To solve these two problems, this article establishes a model for solving the kth discriminant component, which first transforms the original model of Fisher discriminant problem in linear discriminant analysis by using a diagonal estimated matrix for the within-class variance in place of the original within-class covariance matrix, which overcomes the singularity problem of the matrix and projects it to an orthogonal projection space to remove its orthogonal constraints, and subsequently an lq norm regularization term is added to enhance its interpretability for the purpose of dimensionality reduction and classification. Finally, an iterative algorithm for solving the model and a convergence analysis are given, and it is proved that the sequence generated by the algorithm is descended and converges to a local minimum of the problem for any initial value.

Optimality in high-dimensional tensor discriminant analysis

Tensor discriminant analysis is an important topic in tensor data analysis. However, given the many proposals for tensor discriminant analysis methods, there lacks a systematic theoretical study, especially concerning optimality. We fill this gap by providing the minimax lower bounds for the estimation and prediction errors under the tensor discriminant analysis model coupled with the sparsity assumption. We further show that one existing high-dimensional tensor discriminant analysis estimator has matching upper bounds, and is thus optimal. Our results apply to tensors with arbitrary orders and ultra-high dimensions. If one focuses on one-way tensors (i.e., vectors), our results further provide strong theoretical justifications for several popular sparse linear discriminant analysis methods. Numerical studies are also presented to support our theoretical results.

L₂,₁-Norm Regularized Robust and Sparse Linear Discriminant Analysis via an Alternating Direction Method of Multipliers

Recently, an absolute value inequalities discriminant analysis criterion with robustness and sparseness for supervised dimensionality reduction was studied. However, it obtains discriminant directions one by one through greedy search, which makes the sparseness of multiple discriminant directions unexplainable. In addition, it relaxes the original problem into a series of linear programming problems which makes it time consuming. In this paper, we construct a novel linear discriminant analysis with robustness and sparseness jointly through the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">L</i> <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sub> -norm and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">L</i> <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2,1</sub> -norm. The proposed approach obtains all the discriminant directions simultaneously, and rather than solving linear programming problems, it is solved by a more effective alternating direction method of multipliers. The effectiveness of the proposed method is supported by preliminary experimental results on two artificial datasets, some benchmark datasests and two face image datasets.

Adverse effect signature extraction and prediction for drugs treating COVID-19.

Given the considerable cost of drug discovery, drug repurposing is becoming attractive as it can effectively shorten the development timeline and reduce the development cost. However, most existing drug-repurposing methods omitted the heterogeneous health conditions of different COVID-19 patients. In this study, we evaluated the adverse effect (AE) profiles of 106 COVID-19 drugs. We extracted four AE signatures to characterize the AE distribution of 106 COVID-19 drugs by non-negative matrix factorization (NMF). By integrating the information from four distinct databases (AE, bioassay, chemical structure, and gene expression information), we predicted the AE profiles of 91 drugs with inadequate AE feedback. For each of the drug clusters, discriminant genes accounting for mechanisms of different AE signatures were identified by sparse linear discriminant analysis. Our findings can be divided into three parts. First, drugs abundant with AE-signature 1 (for example, remdesivir) should be taken with caution for patients with poor liver, renal, or cardiac functions, where the functional genes accumulate in the RHO GTPases Activate NADPH Oxidases pathway. Second, drugs featuring AE-signature 2 (for example, hydroxychloroquine) are unsuitable for patients with vascular disorders, with relevant genes enriched in signal transduction pathways. Third, drugs characterized by AE signatures 3 and 4 have relatively mild AEs. Our study showed that NMF and network-based frameworks contribute to more precise drug recommendations.

A supervised discriminant data representation: application to pattern classification

The performance of machine learning and pattern recognition algorithms generally depends on data representation. That is why, much of the current effort in performing machine learning algorithms goes into the design of preprocessing frameworks and data transformations able to support effective machine learning. The method proposed in this work consists of a hybrid linear feature extraction scheme to be used in supervised multi-class classification problems. Inspired by two recent linear discriminant methods: robust sparse linear discriminant analysis (RSLDA) and inter-class sparsity-based discriminative least square regression (ICS_DLSR), we propose a unifying criterion that is able to retain the advantages of these two powerful methods. The resulting transformation relies on sparsity-promoting techniques both to select the features that most accurately represent the data and to preserve the row-sparsity consistency property of samples from the same class. The linear transformation and the orthogonal matrix are estimated using an iterative alternating minimization scheme based on steepest descent gradient method and different initialization schemes. The proposed framework is generic in the sense that it allows the combination and tuning of other linear discriminant embedding methods. According to the experiments conducted on several datasets including faces, objects, and digits, the proposed method was able to outperform competing methods in most cases.

Comparison between visual assessments and different variants of linear discriminant analysis to the classification of Raman patterns of inkjet printer inks

Inkjet printers are devices that are frequently used in illicit activities like threats, extorsions, or the production of counterfeited currency. In these cases, scientific information may play a critical role to develop investigative leads and assist investigators with the designation of potential printer source candidates. Raman spectroscopy was used to analyze the three main colored dot components (cyan, magenta, and yellow) of inkjet printed documents. Three variants of linear discriminant analysis (LDA) were evaluated on 231 Raman spectra of a set of 11 inkjet printer ink samples, which were previously compared visually. The three LDA variants were: 1) Principal component analysis (PCA) followed by LDA, 2) Partial least square discriminant analysis (PLSDA) and 3) sparse LDA. The classification accuracy of the selected classifiers was evaluated as well as their detection sensitivity to the level of detail that analysts should typically use during spectral visual comparisons. These evaluations were made for each dot color and also for their joint use. Results showed that although spectral visual comparisons are still superior to differentiate Raman spectra on the basis of minor peaks, sparse LDA provided the highest classification potential (i.e., highest accuracy) for individual colors, and that the three methods performed equivalently when the spectral data from the three colors were combined. It was also determined that baseline correction was not a factor that affected the performance of the used classifiers, but it was found that normalization was a necessary step prior to data analysis.

Discriminative feature learning for blade icing fault detection of wind turbine

The early detection of blade icing is gaining increasing attention due to its importance in guaranteeing wind turbine safety and operation efficiency. In this study, a wind turbine icing fault detection method based on discriminative feature learning is proposed. First, a stacked autoencoder (SAE) is trained to generate representations, which utilizes a large amount of normal operating data, as well as time series correlation information. Second, discriminative features are obtained by combining the original data, SAE-extracted features, and the residual vector. Third, the sparse linear discriminant analysis is performed on the discriminative features to achieve simultaneous feature selection and dimension reduction. Finally, the wind turbine operation status is examined using the learned discriminative feature. The proposed discriminative feature learning-based fault detection scheme is tested on a benchmark wind turbine icing dataset. Results of the comparative trial verify the feasibility and superiority of the proposed method.

Multiband entropy-based feature-extraction method for automatic identification of epileptic focus based on high-frequency components in interictal iEEG

Presurgical investigations for categorizing focal patterns are crucial, leading to localization and surgical removal of the epileptic focus. This paper presents a machine learning approach using information theoretic features extracted from high-frequency subbands to detect the epileptic focus from interictal intracranial electroencephalogram (iEEG). It is known that high-frequency subbands (>80 Hz) include important biomarkers such as high-frequency oscillations (HFOs) for identifying epileptic focus commonly referred to as the seizure onset zone (SOZ). In this analysis, the multi-channel interictal iEEG signals were splitted into segments and each segment was decomposed into multiple high-frequency subbands. The different types of entropy were calculated for each of the subbands and the sparse linear discriminant analysis (sLDA) was applied to select the prominent entropy features. Due to the imbalance of SOZ and non-SOZ channels in iEEG data, the use of machine learning techniques is always tricky. To deal with the imbalanced learning problem, an adaptive synthetic oversampling approach (ADASYN) with radial basis function kernel-based SVM was used to detect the focal segments. Finally, the epileptic focus was identified based on detection of focal segments on SOZ and non-SOZ channels. Eight patients were examined to observe the efficiency of the automatic detector. The experimental results and statistical tests indicate that the proposed automatic detector can identify the epileptic focus accurately and efficiently.

Sparse Ensemble Machine Learning to Improve Robustness of Long-Term Decoding in iBMIs.

This paper presents a novel sparse ensemble based machine learning approach to enhance robustness of intracortical Brain Machine Interfaces (iBMIs) in the face of non-stationary distribution of input neural data across time. Each classifier in the ensemble is trained on a randomly sampled (with replacement) set of input channels. These sparse connections ensure that with a high chance, few of the base classifiers should be less affected by the variations in some of the recording channels. We have tested the generality of this technique on different base classifiers - linear discriminant analysis (LDA), support vector machine (SVM), extreme learning machine (ELM) and multilayer perceptron (MLP). Results show decoding accuracy improvements of up to ≈21 %, 13%, 19%, 10% in non-human primate (NHP) A and 7%, 9%, 7%, 9% in NHP B across test days while using the sparse ensemble approach over a single classifier model for LDA, SVM, ELM and MLP algorithms respectively. Furthermore, improvements of up to ≈7(14)%, 8(15)%, 9(19)%, 7(15)% in NHP A and 8(15)%, 12(20)%, 15(23)%, 12(19)% in NHP B over Random Forest (Long-short Term Memory) have been obtained by sparse ensemble LDA, SVM, ELM, MLP respectively.

Prediction and estimation consistency of sparse multi-class penalized optimal scoring

Sparse linear discriminant analysis via penalized optimal scoring is a successful tool for classification in high-dimensional settings. While the variable selection consistency of sparse optimal scoring has been established, the corresponding prediction and estimation consistency results have been lacking. We bridge this gap by providing probabilistic bounds on out-of-sample prediction error and estimation error of multi-class penalized optimal scoring allowing for diverging number of classes.

Major depressive disorder identification by referenced multiset canonical correlation analysis with clinical scores.

A novel method based on multiset canonical correlation analysis (mCCA) and linear discriminant analysis (LDA) is presented to identify the major depressive disorder (MDD). The new method comprises two parts, namely, the mCCA-rreg and sparse LDA models. The mCCA-rreg model extends the classical canonical correlation model to calculate functional connections by restricting the references to a reference space and adding a spatial regularization term. The reference space is used to ensure that the model extracts important components first from several datasets simultaneously by decreasing the importance of the components in which we are uninterested. The spatial regularization term helps in avoiding the multicollinearity and overfitting problems under the low signal-to-noise ratio circumstance. The sparse LDA model extends the classical LDA model to extract a small subset of discriminative classification features by fusing clinical scores. In the real data experiment, we extract two functional connection modes from 45 subjects by the mCCA-rreg model. Then, we construct classifiers to identify the patients with MDD based on the connections selected by the sparse LDA model. The best accuracy is higher than 95%. The results show that the mCCA-rreg model can retrieve the important components characterized by a preassigned reference space and exclude the noise or components of no interest. The sparse LDA model can extract discriminative classification features related to clinical scores.

Complexity and Applications of the Homotopy Principle for Uniformly Constrained Sparse Minimization

In this paper, we investigate the homotopy path related to $$\ell _{1}$$ -norm minimization problems with $$\ell _{\infty }$$ -norm constraints. We establish an enhanced upper bound on the number of linear segments in the path and provide an example showing that the number of segments is exponential in the number of variables in the worst case. We also use the homotopy framework to develop grid independent (cross-)validation schemes for sparse linear discriminant analysis and classification that make use of the entire path. Several numerical and statistical examples illustrate the applicability of the framework.

Brain connectivity alterations in early psychosis: from clinical to neuroimaging staging

Early in the course of psychosis, alterations in brain connectivity accompany the emergence of psychiatric symptoms and cognitive impairments, including processing speed. The clinical-staging model is a refined form of diagnosis that places the patient along a continuum of illness conditions, which allows stage-specific interventions with the potential of improving patient care and outcome. This cross-sectional study investigates brain connectivity features that characterize the clinical stages following a first psychotic episode. Structural brain networks were derived from diffusion-weighted MRI for 71 early-psychosis patients and 76 healthy controls. Patients were classified into stage II (first-episode), IIIa (incomplete remission), IIIb (one relapse), and IIIc (two or more relapses), according to the course of the illness until the time of scanning. Brain connectivity measures and diffusion parameters (fractional anisotropy, apparent diffusion coefficient) were investigated using general linear models and sparse linear discriminant analysis (sLDA), studying distinct subgroups of patients who were at specific stages of early psychosis. We found that brain connectivity impairments were more severe in clinical stages following the first-psychosis episode (stages IIIa, IIIb, IIIc) than in first-episode psychosis (stage II) patients. These alterations were spatially diffuse but converged on a set of vulnerable regions, whose inter-connectivity selectively correlated with processing speed in patients and controls. The sLDA suggested that relapsing-remitting (stages IIIb, IIIc) and non-remitting (stage IIIa) patients are characterized by distinct dysconnectivity profiles. Our results indicate that neuroimaging markers of brain dysconnectivity in early psychosis may reflect the heterogeneity of the illness and provide a connectomics signature of the clinical-staging model.

Robust Sparse Linear Discriminant Analysis

Linear discriminant analysis (LDA) is a very popular supervised feature extraction method and has been extended to different variants. However, classical LDA has the following problems: 1) The obtained discriminant projection does not have good interpretability for features; 2) LDA is sensitive to noise; and 3) LDA is sensitive to the selection of number of projection directions. In this paper, a novel feature extraction method called robust sparse linear discriminant analysis (RSLDA) is proposed to solve the above problems. Specifically, RSLDA adaptively selects the most discriminative features for discriminant analysis by introducing the $l_{2,1}$ norm. An orthogonal matrix and a sparse matrix are also simultaneously introduced to guarantee that the extracted features can hold the main energy of the original data and enhance the robustness to noise, and thus RSLDA has the potential to perform better than other discriminant methods. Extensive experiments on six databases demonstrate that the proposed method achieves the competitive performance compared with other state-of-the-art feature extraction methods. Moreover, the proposed method is robust to the noisy data.

Graph-based sparse linear discriminant analysis for high-dimensional classification

Linear discriminant analysis (LDA) is a well-known classification technique that enjoyed great success in practical applications. Despite its effectiveness for traditional low-dimensional problems, extensions of LDA are necessary in order to classify high-dimensional data. Many variants of LDA have been proposed in the literature. However, most of these methods do not fully incorporate the structure information among predictors when such information is available. In this paper, we introduce a new high-dimensional LDA technique, namely graph-based sparse LDA (GSLDA), that utilizes the graph structure among the features. In particular, we use the regularized regression formulation for penalized LDA techniques, and propose to impose a structure-based sparse penalty on the discriminant vector β. The graph structure can be either given or estimated from the training data. Moreover, we explore the relationship between the within-class feature structure and the overall feature structure. Based on this relationship, we further propose a variant of our proposed GSLDA to utilize effectively unlabeled data, which can be abundant in the semi-supervised learning setting. With the new regularization, we can obtain a sparse estimate of β and more accurate and interpretable classifiers than many existing methods. Both the selection consistency of β estimation and the convergence rate of the classifier are established, and the resulting classifier has an asymptotic Bayes error rate. Finally, we demonstrate the competitive performance of the proposed GSLDA on both simulated and real data studies.