Multiclass Linear Discriminant Analysis Research Articles

On the Equivalence of Linear Discriminant Analysis and Least Squares Regression.

Studying the relationship between linear discriminant analysis (LDA) and least squares regression (LSR) is of great theoretical and practical significance. It is well-known that the two-class LDA is equivalent to an LSR problem, and directly casting multiclass LDA as an LSR problem, however, becomes more challenging. Recent study reveals that the equivalence between multiclass LDA and LSR can be established based on a special class indicator matrix, but under a mild condition which may not hold under the scenarios with low-dimensional or oversampled data. In this article, we show that the equivalence between multiclass LDA and LSR can be established based on arbitrary linearly independent class indicator vectors and without any condition. In addition, we show that LDA is also equivalent to a constrained LSR based on the data-dependent indicator vectors. It can be concluded that under exactly the same mild condition, such two regressions are both equivalent to the null space LDA method. Illuminated by the equivalence of LDA and LSR, we propose a direct LDA classifier to replace the conventional framework of LDA plus extra classifier. Extensive experiments well validate the above theoretic analysis.

Discriminant analysis using MRI asymmetry indices and cognitive scores of women with temporal lobe epilepsy or schizophrenia.

This study aims to assess the diagnostic power of brain asymmetry indices and neuropsychological tests for differentiating mesial temporal lobe epilepsy (MTLE) and schizophrenia (SCZ). We studied a total of 39 women including 13 MTLE, 13 SCZ, and 13 healthy individuals (HC). A neuropsychological test battery (NPT) was administered and scored by an experienced neuropsychologist, and NeuroQuant (CorTechs Labs Inc., San Diego, California) software was used to calculate brain asymmetry indices (ASI) for 71 different anatomical regions of all participants based on their 3D T1 MR imaging scans. Asymmetry indices measured from 10 regions showed statistically significant differences between the three groups. In this study, a multi-class linear discriminant analysis (LDA) model was built based on a total of fifteen variables composed of the most five significantly informative NPT scores and ten significant asymmetry indices, and the model achieved an accuracy of 87.2%. In pairwise classification, the accuracy for distinguishing MTLE from either SCZ or HC was 94.8%, while the accuracy for distinguishing SCZ from either MTLE or HC was 92.3%. The ability to differentiate MTLE from SCZ using neuroradiological and neuropsychological biomarkers, even within a limited patient cohort, could make a substantial contribution to research in larger patient groups using different machine learning techniques.

Using normalised difference vegetation index in classification and agroecological zoning of spring row crops

Remote sensing is an important branch of modern science and technology with various applications in different branches of life sciences. Its application in agriculture is focused mainly on crop monitoring and yield prediction. However, the value of remote sensing in the systems of automated crop mapping and agroecological zoning of plant species is increasing. The main purpose of this study is to establish the possibility of using normalised difference vegetation index in the main spring row crops, namely maize, soybeans, sunflower, to precisely classify the fields with each crop, and to evaluate the best agroecological zones for their cultivation in rainfed conditions in Ukraine. The study was carried out using the data on the normalised difference vegetation index for the period May – November 2018 from 750 fields and experimental plots, randomly scattered over the territory of Ukraine with equal representation by every administrative district of the country. The index values were calculated using combined Landsat-8 and Sentinel-2 images, with further generalisation for every crop and region. Multiclass linear discriminant analysis and canonical discriminant analysis were applied to determine whether it is possible to distinguish between the studied crops using the values of the normalised difference vegetation index as the only input. As a result, it was established that the best zone for crop cultivation is the west of the country: NDVI values for the growing season averaged to 0.34 for sunflower, 0.36 for soybeans, and 0.36 for maize, respectively. The worst growing conditions, based on the lowest NDVI values, were observed in the east for sunflower (0.26) and maize (0.25), but the minimum NDVI for soybeans (0.27) was observed in the south. Regarding the classification problem, it was found that the highest importance for the classification of crops is attributed to the values of the normalised difference vegetation index, recorded in August. The supervised learning using canonical discriminant function resulted in mediocre predictive performance of the multiple linear function with general classification accuracy of 56.5%. The best accuracy of classification was achieved for sunflower (70.4%), while it is difficult to distinguish between maize and soybeans because these crops have quite similar intra-seasonal dynamics of the vegetation index (classification accuracy was 46.8% and 52.4%, respectively; the total number of incorrectly predicted samples in the “maize-soybeans” group was 134 or 26.8%). The main limitation of this study is its single year basis, notwithstanding the fact that the year of the study was characterized as a typical one for most territory of Ukraine in terms of meteorological conditions. Therefore, more studies are required to clarify the possibility of a classification between maize and soybeans based on remote sensing data.

SAT587 Methylation-based Machine Learning Classifiers Discriminate Sellar Tumors By Lobe Origin Using Liquid Biopsy Or Surgical Specimens

Abstract Disclosure: G.A. Herrgott: None. K.P. Asmaro: None. M. Wells: None. K. Nelson: None. B. Thomas: None. L.A. Hasselbach: None. A. Transou: None. S. Cazacu: None. K.M. Tundo: None. S. Nadimidla: None. L. Scarpace: None. J. Barnholtz-Sloan: None. A.E. Sloan: None. W.R. Selman: None. A.C. deCarvalho: None. A. Mukherjee: None. A.M. Robin: None. I.Y. Lee: None. J. Craig: None. S. Kalkanis: None. J. Snyder: None. T. Walbert: None. J. Rock: None. H. Noushmehr: None. A.B. Castro: None. Background: The differential diagnosis of challenging sellar tumor cases can be inconclusive through imaging features and could benefit from noninvasive diagnostic approaches, such as liquid biopsy (LB). Similar to tissue, LB specimens carry tumor-specific DNA methylation signatures amenable to the construction of accurate machine learning models able to discriminate CNS tumors. We aimed to develop methylation-based classifiers which classify sellar tumors by lobe of origin, using either LB or tumor tissue specimens. Methodology: We analyzed the DNA methylome (EPIC array) of tumor tissue (T) and LB specimens from adult patients with tumors representing each of the three pituitary lobes (Anterior: T=177; LB=37; Intermediate: T= 7; LB: 10 and Posterior: T=44, LB=2 cases). Using the most variably methylated CpG probes derived from the unsupervised variance-based analyses across tumors from different lobes, we applied multi-class linear discriminant analysis to construct machine learning models to classify sellar tumor tissue and/or LB specimens. Results: We generated classifiers based on lobe-specific methylation signatures that were able to discriminate across sellar tumors either using tissue and/or LB specimens (500 and 600 CpGs, respectively) with observed accuracies of ∼99% across independent validation. DISCUSSION/CONCLUSION: Our findings suggest that methylation-based classifiers constitute an accurate diagnostic approach to discriminate sellar tumors according to the lobe origin, either pre-surgically through a blood draw or through surgical tumor specimens. These classifiers are objective approaches that could complement imaging and pathology reports for an accurate diagnosis of inconclusive cases, ultimately leading to optimal management of the patients with these diseases. Presentation: Saturday, June 17, 2023

Adaptive classification helps hybrid visual brain computer interface systems handle non‐stationary cortical signals

AbstractThe classifier efficiency of the brain‐computer interface systems is significantly impacted by the non‐stationarity of electroencephalogram (EEG) signals. We propose an adaptive variant of the linear discriminant analysis (LDA) classifier as a solution to this problem. This classifier constantly adjusts its parameters to account for the most recent EEG data. In this study, the authors will update the mean values as well as the covariance matrix of each class pair. Visually evoked cortical potential datasets are used to check how well the proposed classifier performs. The authors prove that the proposed adaptive LDA performs much better than both static multiclass LDA and adaptive PMean LDA.

Discriminant Analysis of Water Inrush Sources in the Weibei Coalfield, Shaanxi Province, China

Water inrush disasters in mining areas are one of the most serious geological disasters in coal mining. The purpose of this study is to study the establishment of a water chemical database and water inrush source discrimination model in the Weibei coalfield to provide the basis for regional hydrogeological conditions for future mining under pressure in the Weibei area, as well as a basis for the rapid identification of water inrush sources in the Weibei coalfield. In this paper, a conventional hydrochemical and trace element discrimination model for mine water inrush was established, and the hydrochemical characteristic files of the entire mining area were integrated. Based on 10 indicators, three hydrochemical discrimination models of rock stratum aquifers were established. Through the Mahalanobis distance test, it was found that the six selected variables, K+ + Na+, Mg2+, NH4+, Cl−, SO42−, and pH, have significant discrimination ability and good effect and can effectively distinguish the three main water inrush aquifers in the Weibei mining area. Then, the clustering stepwise discriminant analysis method was used to select 24 water samples and 14 trace element indicators from the conventional water chemistry test results. Based on principal component analysis, a principal component analysis discriminant model of trace elements was established for the four main aquifers. The accuracy and misjudgment rate of the Bayes multi-class linear discriminant using conventional ions as explanatory variables were 64.3% and 35.7%, respectively, showing a poor discriminant effect. On this basis, seven characteristic trace elements were analyzed according to Bayes multi-class linear discriminant analysis, the mutual influence and restriction relationship regarding the migration of these seven trace elements in the groundwater system of the mining area was determined, and the modified Bayes multi-class linear discriminant analysis model of trace elements for the water inrush source was established, which was more accurate than the conventional ion Bayes multi-class linear discriminant analysis model. The accuracy rate reached 92.9%. This research is of great significance for mine water-source identification and water-inrush prevention guidance.

Dynamic Reweighted Domain Adaption for Cross-Domain Bearing Fault Diagnosis

Certain progress has been made in fault diagnosis under cross-domain scenarios recently. Most researchers have paid almost all their attention to promoting domain adaptation in a common space. However, several challenges that will cause negative transfer have been ignored. In this paper, a reweighting method is proposed to overcome this difficulty from two aspects. First, extracted features differ greatly from one another in promoting positive transfer, and measuring the difference is important. Measured by conditional entropy, the weight of adversarial losses for those well aligned features are reduced. Second, the balance between domain adaptation and class discrimination greatly influences the transferring task. Here, a dynamic weight strategy is adopted to compute the balance factor. Consideration is made from the perspective of maximum mean discrepancy and multiclass linear discriminant analysis. The first item is supposed to measure the degree of the domain adaptation between source and the target domain, and the second is supposed to show the classification performance of the classifier on the learned features in the current training epoch. Finally, extensive experiments on several bearing fault diagnosis datasets are conducted. The performance shows that our model has an obvious advantage compared with other common transferring algorithms.

Multi-class LDA classifier and CNN feature extraction for student performance analysis during Covid-19 pandemic

In our modern world, education is essential for developing high moral values and excellence in individuals. But the spread of Covid-19 widely affects the student’s education, the majority of students have continued their education via online learning platforms. The academic performance of students has been sluggish across the globe during this pandemic. This problem is solved using a multiclass Linear Discriminant Analysis (LDA) and Convolutional Neural Network (CNN) model which predicts the student learning rate and behavior. This research aims to classify the students’ performance into low, medium, and high grades in order to assist tutors in predicting the low-ranking students. The student data log is collected from the Kaggle student performance analysis dataset and pre-processed to remove the noise and non-redundance data. By analyzing the pre-processed data, the CNN extracts feature that are based on student interest and subjective pattern sequences. Then extracted features are filtered by the Minimum Redundancy Maximum Relevance(mRMR) method. mRMR selects the best features and dilutes the least one which handles each feature separately. The feature weights are measured by Stochastic Gradient Descent (SGD) and updated for better feature learning by CNN. At the last stage, the Multi-class LDA classifier evaluates the result into categorized classes. Based on the prediction, the tutors can easily find the low ranks of students who need a high preference for improving their academic performance. Experiments showed that the proposed model achieves greater accuracy (96.5%), precision (094), recall (092), F-score (095), and requires less computation time than existing methods.

Online multiclass EEG feature extraction and recognition using modified convolutional neural network method

<p>Many techniques have been introduced to improve both brain-computer interface (BCI) steps: feature extraction and classification. One of the emerging trends in this field is the implementation of deep learning algorithms. There is a limited number of studies that investigated the application of deep learning techniques in electroencephalography (EEG) feature extraction and classification. This work is intended to apply deep learning for both stages: feature extraction and classification. This paper proposes a modified convolutional neural network (CNN) feature extractorclassifier algorithm to recognize four different EEG motor imagery (MI). In addition, a four-class linear discriminant analysis (LDR) classifier model was built and compared to the proposed CNN model. The paper showed very good results with 92.8% accuracy for one EEG four-class MI set and 85.7% for another set. The results showed that the proposed CNN model outperforms multi-class linear discriminant analysis with an accuracy increase of 28.6% and 17.9% for both MI sets, respectively. Moreover, it has been shown that majority voting for five repetitions introduced an accuracy advantage of 15% and 17.2% for both EEG sets, compared with single trials. This confirms that increasing the number of trials for the same MI gesture improves the recognition accuracy</p>

Environmental Conditions Associated with Horizontal Convective Rolls, Cellular Convection, and No Organized Circulations

AbstractTypical environmental conditions associated with horizontal convective rolls (HCRs) and cellular convection have been known for over 50 years. Yet our ability to predict whether HCRs, cellular convection, or no discernable organized (null) circulation will occur within a well-mixed convective boundary layer based upon easily observed environmental variables has been limited. Herein, a large database of 50 cases each of HCR, cellular convection, and null events is created that includes observations of mean boundary layer wind and wind shear, boundary layer depth; surface observations of wind, temperature, and relative humidity; and estimates of surface sensible heat flux. Results from a multiclass linear discriminant analysis applied to these data indicate that environmental conditions can be useful in predicting whether HCRs, cellular convection, or no circulation occurs, with the analysis identifying the correct circulation type on 72% of the case days. This result is slightly better than using a mean convective boundary layer (CBL) wind speed of 6 m s−1 to discriminate between HCRs and cells. However, the mean CBL wind speed has no ability to further separate out cases with no CBL circulation. The key environmental variables suggested by the discriminant analysis are mean sensible heat flux, friction velocity, and the Obukhov length.

Multiparameter Space Decision Voting and Fusion Features for Facial Expression Recognition.

Obtaining a valid facial expression recognition (FER) method is still a research hotspot in the artificial intelligence field. In this paper, we propose a multiparameter fusion feature space and decision voting-based classification for facial expression recognition. First, the parameter of the fusion feature space is determined according to the cross-validation recognition accuracy of the Multiscale Block Local Binary Pattern Uniform Histogram (MB-LBPUH) descriptor filtering over the training samples. According to the parameters, we build various fusion feature spaces by employing multiclass linear discriminant analysis (LDA). In these spaces, fusion features composed of MB-LBPUH and Histogram of Oriented Gradient (HOG) features are used to represent different facial expressions. Finally, to resolve the inconvenient classifiable pattern problem caused by similar expression classes, a nearest neighbor-based decision voting strategy is designed to predict the classification results. In experiments with the JAFFE, CK+, and TFEID datasets, the proposed model clearly outperformed existing algorithms.

Human action recognition method based on conformal geometric algebra and recurrent neural network

Deep Learning (DL) plays an important role in machine learning and artificial intelligence. DL is widely applied in many fields with high dimensional data, including natural language processing, image recognition. High dimensional data can lead to problems in machine learning such as overfitting, degradation of accuracy. To address these issues, some methods, such as Principal Components Analysis (PCA), principal component regression (PCR), Multi-class Linear Discriminant Analysis (MLDA), were proposed to reduce dimensions of the data and computational complexity simultaneously. The drawback of these methods is that they only work well on data distributed on the plane. In the case of the data distributed on the hyper-sphere, such as objects moving in space, the processing results are not so good as expected. In this paper, we propose the use of Conformal Geometric Algebra (CGA) to extract features and simultaneously reduce the dimensionality of a dataset for human activity recognition using Recurrent Neural Network (RNN). First, human activity data in a 3-dimensional coordinate system is pre-processed and normalized by calculating deviations from the mean coordinate. Next, the data is transformed to vectors in CGA space and its dimensions are reduced to return the feature vectors. Finally, we use the RNN model to train feature vectors. Empirical results performed on the CMU eight actions dataset show that the CGA combined with RNN gave the best test results of 92.5%.

A modified boosted support vector machine to rate banks

PurposeThe purpose of this study is to compare the classification learning ability of our algorithm based on boosted support vector machine (B-SVM), against other classification techniques in predicting the credit ratings of banks. The key feature of this study is the usage of an imbalanced dataset (in the response variable/rating) with a smaller number of observations (number of banks).Design/methodology/approachIn general, datasets in banking sector are small and imbalanced too. In this study, 23 Scheduled Commercial Banks (SCBs) have been chosen (in India), and their corresponding corporate ratings have been collated from the Indian subsidiary of reputed global rating agency. The top management of the rating agency provided 12 input (quantitative) variables that are considered essential for rating a bank within India. In order to overcome the challenge of dataset being imbalanced and having small number of observations, this study uses an algorithm, namely “Modified Boosted Support Vector Machines” (MBSVMs) proposed by Punniyamoorthy Murugesan and Sundar Rengasamy. This study also compares the classification ability of the aforementioned algorithm against other classification techniques such as multi-class SVM, back propagation neural networks, multi-class linear discriminant analysis (LDA) and k-nearest neighbors (k-NN) classification, on the basis of geometric mean (GM).FindingsThe performances of each algorithm have been compared based on one metric—the geometric mean, also known as GMean (GM). This metric typically indicates the class-wise sensitivity by using the values of products. The findings of the study prove that the proposed MBSVM technique outperforms the other techniques.Research limitations/implicationsThis study provides an algorithm to predict ratings of banks where the dataset is small and imbalanced. One of the limitations of this research study is that subjective factors have not been included in our model; the sole focus is on the results generated by the models (driven by quantitative parameters). In future, studies may be conducted which may include subjective parameters (proxied by relevant and quantifiable variables).Practical implicationsVarious stakeholders such as investors, regulators and central banks can predict the credit ratings of banks by themselves, by inputting appropriate data to the model.Originality/valueIn the process of rating banks, the usage of an imbalanced dataset can lessen the performance of the soft-computing techniques. In order to overcome this, the authors have come up with a novel classification approach based on “MBSVMs”, which can be used as a yardstick for such imbalanced datasets. For this purpose, through primary research, 12 features have been identified that are considered essential by the credit rating agencies.

Automated identification of the predominant site of upper airway collapse in obstructive sleep apnoea patients using snore signal

Objective: This study provides a novel approach for an automated system using a machine learning algorithm to predict the predominant site of upper airway collapse into four classes (‘lateral wall’, ‘palate’, ‘tongue-base’ related collapse or ‘multi-level’ site-of-collapse) in obstructive sleep apnoea (OSA) patients from the audio signal recorded during normal sleep. Approach: Snore sounds from 58 patients were recorded simultaneously with full-night polysomnography during sleep with a ceiling mounted microphone. The probable site-of-airway collapse was determined by manual analysis of the shape of the airflow signal during hypopnoea. Time and frequency features of the audio signal were extracted from each hypopnoea event to classify the audio signal into ‘lateral wall’, ‘palate’ and ‘tongue-base’ related collapse according to prior research. The data was divided into two sets. The Learning Set contained the data of the first 45 patients and was used for building the model. The Hidden Set contained the data from the remaining 13 patients and was used for testing the performance of the model. Feature selection was employed to boost the classification performance. The classification was carried out with a multi-class linear discriminant analysis classifier to classify the predominant site-of-collapse for a patient into the four classes. Performance was evaluated by comparing the automatic and manually labelled data based on the predominant site-of-collapse and calculating the accuracy. Main results: The model achieved an overall accuracy on the Hidden Set of 77% for discriminating tongue/non-tongue collapse and an accuracy of 62% accuracy for all site-of-collapse classes. Significance: Our results demonstrate that the audio signal recorded during sleep can successfully identify the site-of-collapse in the upper airway. The additional information regarding the obstruction site may assist clinicians in deciding the most appropriate treatment for OSA.

Identifying the Predominant Site of Upper Airway Collapse in Obstructive Sleep Apnoea Patients Using Snore Signals.

Knowledge regarding the site of airway collapse could help in choosing an appropriate structure-specific or individualized treatment for obstructive sleep apnoea (OSA). We investigated if the audio signal recorded during hypopnoea (partial obstruction) events can predict the site-of-collapse of the upper airway. In this study, we designed an automatic classifier that predicts the predominant site of upper airway collapse for a patient as "lateral wall", "palate", "tongue-based" related collapse or "multi-level" site-of-collapse by processing of the audio signal. The probable site-of-collapse was determined by manual analysis of the shape of the airflow signal during hypopnoea, which has been reported to correlate with the site of collapse. Audio signal was recorded simultaneously with full-night polysomnography during sleep with a ceiling microphone. Various time and frequency features of the audio signal were extracted to classify the audio signal into lateral wall, palate and tongue-base related collapse. We introduced an unbiased process using nested leave-one patient-out cross-validation to choose the optimal features. The classification was carried out with a multi-class linear discriminant analysis classifier. Performance of the proposed model showed that our automatic system can achieve an overall accuracy of 65% for determining the predominant site-of-collapse for all site-of-collapse classes and an accuracy of 80% for classifying tongue/non-tongue related collapse. Our results indicate that the audio signal recorded during sleep can be helpful in identifying the site-of-collapse and therefore could potentially be used as a new tool for deciding appropriate treatment for OSA.

Tree Species Classification Based on Hybrid Ensembles of a Convolutional Neural Network (CNN) and Random Forest Classifiers

In this paper, we evaluate different popular voting strategies for fusion of classifier results. A convolutional neural network (CNN) and different variants of random forest (RF) classifiers were trained to discriminate between 15 tree species based on airborne hyperspectral imaging data. The spectral data was preprocessed with a multi-class linear discriminant analysis (MCLDA) as a means to reduce dimensionality and to obtain spatial–spectral features. The best individual classifier was a CNN with a classification accuracy of 0.73 +/− 0.086. The classification performance increased to an accuracy of 0.78 +/− 0.053 by using precision weighted voting for a hybrid ensemble of the CNN and two RF classifiers. This voting strategy clearly outperformed majority voting (0.74), accuracy weighted voting (0.75), and presidential voting (0.75).

Discriminative Dimension Reduction via Maximin Separation Probability Analysis.

In this paper, we propose a novel discriminative dimension reduction (DR) method, maximin separation probability analysis (MSPA), which maximizes the minimum separation probability of all classes in the reduced low-dimensional subspace. Separation probability is a novel class separability measure, which gives a lower bound of the generalization accuracy for a learned linear classifier in a binary classification problem. The proposed MSPA duly considers the separation of all class pairs in multiclass linear discriminant analysis (LDA) and thus improves the subsequent classification performance. DR via MSPA leads to a nonconvex optimization problem. We develop an algorithm to solve the problem and the global optimal solution can be found by converting the original problem into a series of second-order cone programming problems. A low-computational cost extension and a non-LDA with kernel mapping of MSPA are also provided in this paper. The experimental results on 14 real-world datasets show our methods are superior to other state-of-the-art algorithms in discriminative DR tasks.

Multiclass linear discriminant analysis with ultrahigh-dimensional features.

Within the framework of Fisher's discriminant analysis, we propose a multiclass classification method which embeds variable screening for ultrahigh-dimensional predictors. Leveraging interfeature correlations, we show that the proposed linear classifier recovers informative features with probability tending to one and can asymptotically achieve a zero misclassification rate. We evaluate the finite sample performance of the method via extensive simulations and use this method to classify posttransplantation rejection types based on patients' gene expressions.

Adaptive Hybrid Classifier for Myoelectric Pattern Recognition Against the Interferences of Outlier Motion, Muscle Fatigue, and Electrode Doffing.

Traditional myoelectric prostheses that employ a static pattern recognition model to identify human movement intention from surface electromyography (sEMG) signals hardly adapt to the changes in the sEMG characteristics caused by interferences from daily activities, which hinders the clinical applications of such prostheses. In this paper, we focus on methods to reduce or eliminate the impacts of three types of daily interferences on myoelectric pattern recognition (MPR), i.e., outlier motion, muscle fatigue, and electrode doffing/donning. We constructed an adaptive incremental hybrid classifier (AIHC) by combining one-class support vector data description and multi-class linear discriminant analysis in conjunction with two specific update schemes. We developed an AIHC-based MPR strategy to improve the robustness of MPR against the three interferences. Extensive experiments on hand-motion recognition were conducted to demonstrate the performance of the proposed method. Experimental results show that the AIHC has significant advantages over non-adaptive classifiers under various interferences, with improvements in the classification accuracy ranging from 7.1% to 39% ( ). The additional evaluations on data deviations demonstrate that the AIHC can accommodate large-scale changes in the sEMG characteristics, revealing the potential of the AIHC-based MPR strategy in the development of clinical myoelectric prostheses.

Penalized Orthogonal Iteration for Sparse Estimation of Generalized Eigenvalue Problem

ABSTRACTWe propose a new algorithm for sparse estimation of eigenvectors in generalized eigenvalue problems (GEPs). The GEP arises in a number of modern data-analytic situations and statistical methods, including principal component analysis (PCA), multiclass linear discriminant analysis (LDA), canonical correlation analysis (CCA), sufficient dimension reduction (SDR), and invariant co-ordinate selection. We propose to modify the standard generalized orthogonal iteration with a sparsity-inducing penalty for the eigenvectors. To achieve this goal, we generalize the equation-solving step of orthogonal iteration to a penalized convex optimization problem. The resulting algorithm, called penalized orthogonal iteration, provides accurate estimation of the true eigenspace, when it is sparse. Also proposed is a computationally more efficient alternative, which works well for PCA and LDA problems. Numerical studies reveal that the proposed algorithms are competitive, and that our tuning procedure works well. We demonstrate applications of the proposed algorithm to obtain sparse estimates for PCA, multiclass LDA, CCA, and SDR. Supplementary materials for this article are available online.