Traditional Linear Discriminant Analysis Research Articles

On Mean-Optimal Robust Linear Discriminant Analysis

Linear discriminant analysis (LDA) is widely used for dimensionality reduction under supervised learning settings. Traditional LDA objective aims to minimize the ratio of the squared Euclidean distances that may not perform optimally on noisy datasets. Multiple robust LDA objectives have been proposed to address this problem, but their implementations have two major limitations. One is that their mean calculations use the squared \(\ell_{2}\) -norm distance to center the data, which is not valid when the objective depends on other distance functions. The second problem is that there is no generalized optimization algorithm to solve different robust LDA objectives. In addition, most existing algorithms can only guarantee that the solution is locally optimal rather than globally optimal. In this article, we review multiple robust loss functions and propose a new and generalized robust objective for LDA. Besides, to remove the mean value within data better, our objective uses an optimal way to center the data through learning. As one important algorithmic contribution, we derive an efficient iterative algorithm to optimize the resulting non-smooth and non-convex objective function. We theoretically prove that our solution algorithm guarantees that both the objective and the solution sequences converge to globally optimal solutions at a sub-linear convergence rate. The results of comprehensive experimental evaluations demonstrate the effectiveness of our new method, achieving significant improvements compared to the other competing methods.

Toward Robust Discriminative Projections Learning Against Adversarial Patch Attacks.

As one of the most popular supervised dimensionality reduction methods, linear discriminant analysis (LDA) has been widely studied in machine learning community and applied to many scientific applications. Traditional LDA minimizes the ratio of squared l2 norms, which is vulnerable to the adversarial examples. In recent studies, many l1 -norm-based robust dimensionality reduction methods are proposed to improve the robustness of model. However, due to the difficulty of l1 -norm ratio optimization and weakness on defending a large number of adversarial examples, so far, scarce works have been proposed to utilize sparsity-inducing norms for LDA objective. In this article, we propose a novel robust discriminative projections learning (rDPL) method based on the l1,2 -norm trace-ratio minimization optimization algorithm. Minimizing the l1,2 -norm ratio problem directly is a much more challenging problem than the traditional methods, and there is no existing optimization algorithm to solve such nonsmooth terms ratio problem. We derive a new efficient algorithm to solve this challenging problem and provide a theoretical analysis on the convergence of our algorithm. The proposed algorithm is easy to implement and converges fast in practice. Extensive experiments on both synthetic data and several real benchmark datasets show the effectiveness of the proposed method on defending the adversarial patch attack by comparison with many state-of-the-art robust dimensionality reduction methods.

Improved secure PCA and LDA algorithms for intelligent computing in IoT‐to‐cloud setting

AbstractThe rapid development of new technologies such as artificial intelligence and big data analysis requires the simultaneous development of cloud computing technology. The application of IoT‐to‐cloud setting has been fully applied in various industry sectors, such as sensor‐cloud system which is composed of wireless sensor network and cloud computing technology. With the increasing amount and types of collected data, companies need to reduce the dimension of massive data in cloud servers for obtaining data analysis reports rapidly. Due to frequent cloud server data leaks, companies must adequately protect the privacy of some confidential data. To this end, we designed a dimension reduction method for ciphertext data in the sensor‐cloud system based on the CKKS encryption scheme, principal component analysis (PCA) and linear discriminant analysis (LDA) dimension reduction algorithm. As data cannot be directly calculated using traditional PCA and LDA algorithm after encryption, we add some interactive operations and iterative calculations to replace some steps in traditional algorithms. Finally, we select the classification dataset IRIS which is commonly used in machine learning, and screen out the best encryption and calculation parameters, and efficiently realize the dimension reduction method of ciphertext data through a large number of experiments.

Development of a read-across-derived classification model for the predictions of mutagenicity data and its comparison with traditional QSAR models and expert systems

Mutagenicity is considered an important endpoint from the regulatory, environmental and medical points of view. Due to the wide number of compounds that may be of concern and the enormous expenses (in terms of time, money, and animals) associated with rodent mutagenicity bioassays, this endpoint is a major target for the development of alternative approaches for screening and prediction. The majority of old-aged expert systems and quantitative structure-activity relationship (QSAR) models may show reduced performance over time for their application on newer chemical candidates; thus, researchers constantly try to improve the modeling strategies. In our report, we initially performed traditional classification-based linear discriminant analysis (LDA) QSAR modeling using the benchmark Ames dataset of diverse chemicals (6512 compounds) to recognize the relationship between the molecules and their potential mutagenic behavior. The classical LDA QSAR model is developed from a selected set of 2D descriptors. The LDA QSAR model was developed by using a total of 31 descriptors identified from the analysis of the most discriminating features. Additionally, we have used similarity-derived features obtained from the read-across (RA) to develop an RA-based QSAR model. The developed RA-based LDA QSAR model has better predictivity, transferability, and interpretability compared to the LDA QSAR model, and it uses a very small number of descriptors compared to the classical QSAR model. Different machine learning (ML) models were also developed using the descriptors appearing in the read-across-based LDA QSAR model for comparative studies. We have checked the prediction quality of 216 true external set compounds using the novel similarity-derived RA model. The performance of the OECD toolbox is also compared with the RA-derived LDA QSAR model for a true external set. The current study aimed to explore the significance of the read-across-based algorithm and its application to the most current experimental mutagenicity data to complement already available expert systems.

RamanCMP: A Raman spectral classification acceleration method based on lightweight model and model compression techniques

In recent years, Raman spectroscopy combined with deep learning techniques has been widely used in various fields such as medical, chemical, and geological. However, there is still room for optimization of deep learning techniques and model compression algorithms for processing Raman spectral data. To further optimize deep learning models applied to Raman spectroscopy, in this study time, accuracy, sensitivity, specificity and floating point operations numbers(FLOPs) are used as evaluation metrics to optimize the model, which is named RamanCompact(RamanCMP). The experimental data used in this research are selected from the RRUFF public dataset, which consists of 723 Raman spectroscopy data samples from 10 different mineral categories. In this paper, 1D-EfficientNet adapted to the spectral data as well as 1D-DRSN are proposed to improve the model classification accuracy. To achieve better classification accuracy while optimizing the time parameters, three model compression methods are designed: knowledge distillation using 1D-EfficientNet model as a teacher model to train convolutional neural networks(CNN), proposing a channel conversion method to optimize 1D-DRSN model, and using 1D-DRSN model as a feature extractor in combination with linear discriminant analysis(LDA) model for classification. Compared with the traditional LDA and CNN models, the accuracy of 1D-EfficientNet and 1D-DRSN is improved by more than 20%. The time of the distilled model is reduced by 9680.9s compared with the teacher model 1D-EfficientNet under the condition of losing 2.07% accuracy. The accuracy of the distilled model is improved by 20% compared to the CNN student model while keeping inference efficiency constant. The 1D-DRSN optimized with channel conversion method saves 60% inference time of the original 1D-DRSN model. Feature extraction reduces the inference time of 1D-DRSN model by 93% with 94.48% accuracy. This study innovatively combines lightweight models and model compression algorithms to improve the classification speed of deep learning models in the field of Raman spectroscopy, forming a complete set of analysis methods and laying the foundation for future research.

Semi-supervised LDA pedestrian re-identification algorithm based on K-nearest neighbor resampling

Person re-identification identify a specific person in surveillance network by similarity measurement between images of different camera views. However, existing metric learning based methods suffer from over-fitting problem. To solve this problem, a resampled linear discriminant analysis (LDA) method was proposed based on the statistical and topological characteristics of pedestrian images. This method utilized the k-nearest neighbours to form potential positive sample pairs. The potential positive pairs are used to improve the metric model and generalize the metric model to the test data. By minimizing the inter-class divergence of potential positive sample pairs, a semi-supervised re-sampling LDA person re-identification algorithm was established. It was then tested on the VIPeR, CUHK01 and Market 1501datasets. The results show that the proposed method achieves the best performance compared to some available methods. Especially, the proposed method outplays the best comparison method by 0.6% and 5.76% at rank-1 identification rate on the VIPeR and CUHK01 datasets respectively. At the same time, the improved LDA algorithm has improved the rank-1 identification accuracy of traditional LDA method by 9.36% and 32.11% on these two datasets respectively. Besides, the proposed method is limited to Market-1501 dataset when the test data is of large size.

Rapid Recognition of Different Sources of Heroin Drugs by Using a Hand-Held Near-Infrared Spectrometer Based on a Multi-Layer Extreme Learning Machine Algorithm

Rapid recognition of the sources of drugs can provide some valuable clues and the basis for determining the nature of the case. A novel recognition method was put forward to identify the sources of heroin drugs rapidly and non-destructively by using a hand-held near infrared (NIR) spectrometer and a multi-layer-extreme learning machine (ML-ELM) algorithm. In contrast to traditional linear discriminant analysis (LDA), support vector machine (SVM) and extreme learning machine (ELM) algorithms, the accuracy, sensitivity and specificity were the highest for the proposed ML-ELM algorithm. The prediction accuracy of the ML-ELM algorithm was 25.33, 20.00, 17.33% higher than that of LDA, SVM and ELM algorithm, respectively, for 4 cases. The ML-ELM models for recognizing the different sources of heroin drugs had the best generalization ability and prediction results. The experimental results indicated that the combination of the hand‑held NIR technology and ML-ELM algorithm can recognize the different sources of heroin drugs rapidly, accurately, and non-destructively on the spot.

Analysis of Proposed and Traditional Boosting Algorithm with Standalone Classification Methods for Classifying Gene Expresssion Microarray Data Using a Reject Option

ABSTRACT In medical field, accurate decisions are very important as they risk human lives. decision support system (DSS) plays important role in making accurate decisions and used for classification/prediction. In gene expression analysis, genes are not only inflated by the external environmental conditions but also the expression values of certain genes are affected (like cancer, obesity etc). in this study, various traditional (Support Vector Machine, Decision Trees, and Linear Discriminant Analysis, naïve Bayes, logistic regression, and multilayer perceptron) and proposed methods (combination of traditional with ensemble and probabilistic classifiers) are used in order to perform the classification and prediction analysis. In this study we used the publicly available datasets comprised of Lymphoid, Leukemia and Colon Cancer. The classification performance on Colon dataset with traditional methods was obtained with accuracy (56%) and proposed probabilistic ensemble methods with accuracy (88%). For dataset, Leukemia, the accuracy was obtained using traditional methods (78%) and proposed methods (92%). Similarly, on Lymphoid dataset, the traditional methods yielded accuracy (75%) and proposed methods (87%). The results revealed that proposed methods yielded the improved detection performance. The proposed methods can be used as a better predictor for early diagnosis and improved diagnosis to improve the healthcare systems.

Rapid recognition of different sources of methamphetamine drugs based on hand-held near infrared spectroscopy and multi-layer-extreme learning machine algorithms

The rapid recognition of the sources of the drugs can provide valuable clues and provide the basis for determining the nature of a drug case. Here, a novel recognition method was put forward to identify the source of methamphetamine drugs rapidly and non-destructively by using a hand-held near infrared (NIR) spectrometer and a multi-layer-extreme learning machine (ML-ELM) algorithm. The accuracy, precision, sensitivity, and F-score were higher with the proposed ML-ELM algorithm than in traditional linear discriminant analysis (LDA), extreme learning machine (ELM) classification, and partial least squares (PLS) regression algorithms. The prediction accuracy of ML-ELM algorithm is 25.0%, 15.3% and 18.1% higher than that of LDA, ELM and PLS regression, respectively. The ML-ELM models for recognizing the different sources of methamphetamine drugs had the best generalization ability and prediction results. The experimental results indicated that the combination of hand-held NIR technology and ML-ELM algorithm can recognize the different sources of methamphetamine drugs rapidly, accurately, and non-destructively.

Saliency-Based Multilabel Linear Discriminant Analysis.

Linear discriminant analysis (LDA) is a classical statistical machine-learning method, which aims to find a linear data transformation increasing class discrimination in an optimal discriminant subspace. Traditional LDA sets assumptions related to the Gaussian class distributions and single-label data annotations. In this article, we propose a new variant of LDA to be used in multilabel classification tasks for dimensionality reduction on original data to enhance the subsequent performance of any multilabel classifier. A probabilistic class saliency estimation approach is introduced for computing saliency-based weights for all instances. We use the weights to redefine the between-class and within-class scatter matrices needed for calculating the projection matrix. We formulate six different variants of the proposed saliency-based multilabel LDA (SMLDA) based on different prior information on the importance of each instance for their class(es) extracted from labels and features. Our experiments show that the proposed SMLDA leads to performance improvements in various multilabel classification problems compared to several competing dimensionality reduction methods.

A Layering Linear Discriminant Analysis-Based Fault Diagnosis Method for Grid-Connected Inverter

Grid-connected inverters are the core equipment for connecting marine energy power generation systems to the public electric utility. The variation of current sensor fault severity will make fault samples multimodal. However, linear discriminant analysis assumes that the same fault is independent and identically distributed. To solve this problem, this paper proposes a layering linear discriminant analysis method based on traditional linear discriminant analysis. The proposed method divides the historical fault data based on the sensor fault severity layer-by-layer until the distribution of the same fault category in each subset is very close. Linear discriminant analysis is used to analyze historical fault data in each subgroup, and the kappa coefficient is applied as the basis for ending the training process. A BP neural network is employed to estimate the fault severity during the testing process, and the fault diagnosis sub-model is selected. The proposed method enables the accurate diagnosis of faults with different distributions in the same category and provides an accurate estimate of the sensor’s fault severity degree. The estimated value of the sensor’s fault degree can provide critical information for the maintenance of the equipment and can be used to correct the sensor’s output.

Self-weighted learning framework for adaptive locality discriminant analysis

Linear discriminant analysis (LDA) is one of the most important dimensionality reduction techniques and applied in many areas. However, traditional LDA algorithms aim to capture the global structure from data and ignore the local information. That may lead to the failure of LDA in some real-world datasets which have a complex geometry distribution. Although there are many previous works that focus on preserving the local information, they are all stuck in the same problem that the neighbor relationships of pairwise data points obtained from the original space may not be reliable, especially in the case of heavy noise. Therefore, we proposed a novel self-weighted learning framework, named Self-Weighted Adaptive Locality Discriminant Analysis (SALDA), for locality-aware based dimensionality reduction. The proposed framework can adaptively learn an intrinsic low-dimensional subspace, so that we can explore the better neighbor relationships for samples under the ideal subspace. In addition, our model can automatically learn to assign the weights to data pairwise points within the same class and takes no extra parameters compared to other classical locality-aware methods. At last, the experimental results on both synthetic and real-world benchmark datasets demonstrate the effectiveness and superiority of the proposed algorithm.

Determining the geographical origin of lettuce with data mining applied to micronutrients and soil properties

Lettuce (Lactuca sativa) is the main leafy vegetable produced in Brazil. Since its production is widespread all over the country, lettuce traceability and quality assurance is hampered. In this study, we propose a new method to identify the geographical origin of Brazilian lettuce. The method uses a powerful data mining technique called support vector machines (SVM) applied to elemental composition and soil properties of samples analyzed. We investigated lettuce produced in São Paulo and Pernambuco, two states in the southeastern and northeastern regions in Brazil, respectively. We investigated efficiency of the SVM model by comparing its results with those achieved by traditional linear discriminant analysis (LDA). The SVM models outperformed the LDA models in the two scenarios investigated, achieving an average of 98 % prediction accuracy to discriminate lettuce from both states. A feature evaluation formula, called F–score, was used to measure the discriminative power of the variables analyzed. The soil exchangeable cation capacity, soil contents of low crystalized Al and Zn content in lettuce samples were the most relevant components for differentiation. Our results reinforce the potential of data mining and machine learning techniques to support traceability strategies and authentication of leafy vegetables.

Marginal Fisher Analysis With Polynomial Matrix Function

Marginal fisher analysis (MFA) is a dimensionality reduction method based on a graph embedding framework. In contrast to traditional linear discriminant analysis (LDA), which requires the data to follow a Gaussian distribution, MFA is suitable for non-Gaussian data, and it has better pattern classification ability. However, MFA has the small-sample-size (SSS) problem. This paper aims to solve the small-sample-size problem while increasing the classification performance of MFA. Based on a matrix function dimensionality reduction framework, the criterion of the MFA method is reconstructed by using the polynomials matrix function transformation, and then a new MFA method is proposed, named PMFA (polynomial marginal fisher analysis). The major contributions of the proposed PMFA method are that it solves the small-sample-size problem of MFA, and it can enlarge the distance between marginal sample points of inter-class, so that it can get better pattern classification performance. Experiments on the public face datasets show that PMFA can get a better classification ability than MFA and its improved methods.

Rapid Recognizing the Producing Area of a Tobacco Leaf Using Near-Infrared Technology and a Multi-Layer Extreme Learning Machine Algorithm

A novel recognition method was put forward to identify the producing areas of the flue-cured tobacco leaves rapidly and non-destructively by using a near-infrared (NIR) spectrometer and a multi-layer-extreme learning machine (ML-ELM) algorithm. In contrast to traditional linear discriminant analysis (LDA) and extreme learning machine (ELM) algorithms, the accuracy, sensitivity and specificity were the highest for the proposed ML-ELM algorithm. The ML-ELM models for different producing areas of Yunnan tobacco leaves had the best generalization ability and prediction results. Besides, the above three algorithms were also identified by using the chemical index data. The experimental results indicated that the NIR spectroscopy technology together with ML-ELM algorithm achieved the best prediction performance both using the NIR spectral data and chemical index data. It indicates that the combination of NIR and ML-ELM can recognize different producing areas of Yunnan tobacco leaves rapidly, accurately, and non-destructively.

Deep Regularized Discriminative Network

Traditional linear discriminant analysis (LDA) approach discards the eigenvalues which are very small or equivalent to zero, but quite often eigenvectors corresponding to zero eigenvalues are the important dimensions for discriminant analysis. We propose an objective function which would utilize both the principal as well as nullspace eigenvalues and simultaneously inherit the class separability information onto its latent space representation. The idea is to build a convolutional neural network (CNN) and perform the regularized discriminant analysis on top of this and train it in an end-to-end fashion. The backpropagation is performed with a suitable optimizer to update the parameters so that the whole CNN approach minimizes the within class variance and maximizes the total class variance information suitable for both multi-class and binary class classification problems. Experimental results on four databases for multiple computer vision classification tasks show the efficacy of our proposed approach as compared to other popular methods.

Balanced multiple weighted linear discriminant analysis and its application to visual process monitoring

Abstract Visual process monitoring is important in complex chemical processes. To address the high state separation of industrial data, we propose a new criterion for feature extraction called balanced multiple weighted linear discriminant analysis (BMWLDA). Then, we combine BMWLDA with self-organizing map (SOM) for visual monitoring of industrial operation processes. BMWLDA can extract the discriminative feature vectors from the original industrial data and maximally separate industrial operation states in the space spanned by these discriminative feature vectors. When the discriminative feature vectors are used as the input to SOM, the training result of SOM can differentiate industrial operation states clearly. This function improves the performance of visual monitoring. Continuous stirred tank reactor is used to verify that the class separation performance of BMWLDA is more effective than that of traditional linear discriminant analysis, approximate pairwise accuracy criterion, max–min distance analysis, maximum margin criterion, and local Fisher discriminant analysis. In addition, the method that combines BMWLDA with SOM can effectively perform visual process monitoring in real time.

Nondestructive and rapid grading of tobacco leaves by use of a hand-held near-infrared spectrometer, based on a particle swarm optimization-extreme learning machine algorithm

A nondestructive and rapid method has been put forward to grade tobacco leaves in the paper. The method is based on a combination of a hand-held near-infrared spectrometer and a particle swarm optimization-extreme learning machine algorithm. Firstly, the spectral data of the training samples are collected directly from the tobacco leaves nondestructively by using a hand-held near infrared spectrometer without any pretreatment. Secondly, the training models of different classes are built using particle swarm optimization-extreme learning machine algorithm. Finally, the classes of test samples can be predicted by using the developed models. Besides, the classification results of particle swarm optimization-extreme learning machine algorithm are also compared with that of the traditional linear discriminant analysis, support vector machine, and extreme learning machine algorithms, respectively. The experimental results show the classification accuracy of the particle swarm optimization-extreme learning machine algorithm is comparable after the parameter optimization. It indicates that the interplay between the hand-held near-infrared spectroscopy technology and particle swarm optimization-extreme learning machine algorithm will provide a novel classification method for grading tobacco leaves in the purchasing process on the spot.

Improvement motor imagery EEG classification based on sparse common spatial pattern and regularized discriminant analysis

BackgroundThe classification of psychological tasks such as motor imagery based on electroencephalography (EEG) signals is an essential issue in the brain computer interface (BCI) system. The feature extraction is an important issue for improving classification accuracy of BCI system. New MethodFor extracting discriminative features, common spatial pattern (CSP) is an effective feature extraction method. However, features extracted by CSP are dense, and even feature patterns are repeatedly selected in the feature space. A sparse CSP algorithm is proposed, which embeds the sparse techniques and iterative search into the CSP. To improve the classification performance, two regularization parameters are added to the traditional linear discriminant analysis (LDA). ResultsThe sparse CSP algorithm can select several channels of EEG signals with the most obvious features. The improved regularized discriminant analysis is used to solve the singularity problem and improve the feature classification accuracy.Comparison with Existing Method(s): The proposed algorithm was evaluated by the data set I of the IVth BCI competition and our dataset. The experimental results of the BCI competition dataset show that accuracy of the improved algorithm is 10.75 % higher than that of the traditional algorithm. Comparing with the currently existing methods for the same data, it also shows excellent classification performance. The effectiveness of the improved algorithm is also shown in experiments on our dataset. ConclusionsIt sufficiently proves that the improved algorithm proposed in this paper improves the classification performance of motor intent recognition.

Seismic landslide susceptibility assessment based on ADASYN-LDA model

Seismic landslide susceptibility (SLS) assessment can be used to estimate the susceptibility of landslides induced by an earthquake, which has great significance for emergency measure making and land use planning to mitigate the landslide hazard and risk. At present, one of the key problems in SLS evaluation is that the sample is extremely unbalanced, that is, the number of landslide units is usually far less than that of non-landslide units, which easily leads to poor prediction performance of SLS model. In addition, when using high-resolution mapping units, the sample size greatly increases, and thus the training time cost is considerable especially for complicated nonlinear artificial intelligence such as the Support Vector Machine and the Neural Networks. In view of this, this paper proposes an SLS model (called ADASYN-LDA) combining ADASYN sample method and linear discriminant analysis (LDA), and evaluates it using 2014 Ludian earthquake event. The results show that the two ADASYN-LDA models are highly efficient in training, and have good prediction performance with the high AUC of 0.8737 and 0.8535 respectively, and with the balanced accuracy of 0.7950 and 0.6174 respectively. Compared with the traditional LDA model without ADASYN method, ADASYN-LDA model improves the results of the SLS assessment affected by the sample imbalance. The further evaluation is needed in other earthquake events, and the further improvement can be made with more landslide conditioning factors and seismic landslide data set.