Unsupervised Linear Discriminant Analysis Research Articles

Neuro or Symbolic? Fine-tuned Transformer with Unsupervised LDA Topic Clustering for Text Sentiment Analysis

For text sentiment analysis, state-of-the-art neural language models have demonstrated promising performance. However, they lack interpretability, require vast volumes of annotated data, and are typically specialized for tasks. In this paper, we explore a connection between fine-tuned Transformer models and unsupervised LDA approach to cope with text sentiment analysis tasks, inspired by the concept of Neuro-symbolic AI. The Transformer and LDA models are combined as a feature extractor to extract the hidden representations of the input text sequences. Subsequently, we employ a feedforward network to forecast various sentiment analysis tasks, such as multi-label emotion prediction, dialogue quality prediction, and nugget detection. Our proposed method obtains the best results in the NTCIR-16 dialogue evaluation (DialEval-2) task, as well as cutting-edge results in emotional intensity prediction using the Ren_CECps corpus. Extensive experiments show that our proposed method is highly explainable, cost-effective in training, and superior in terms of accuracy and robustness.

Discriminative Projected Clustering via Unsupervised LDA.

This work focuses on the projected clustering problem. Specifically, an efficient and parameter-free clustering model, named discriminative projected clustering (DPC), is proposed for simultaneously low-dimensional and discriminative projection learning and clustering, from the perspective of least squares regression. The proposed DPC, a constrained regression model, aims at finding both a transformation matrix and a binary indicator matrix to minimize the sum-of-squares error. Theoretically, a significant conclusion is drawn and used to reveal the connection between DPC and linear discriminant analysis (LDA). Experimentally, experiments are conducted on both toy and real-world data to validate the effectiveness and efficiency of DPC; experiments are also conducted on hyperspectral images to further verify its practicability in real-world applications. Experimental results demonstrate that DPC achieves comparable or superior results to some state-of-the-art clustering methods.

An Effective Clustering Optimization Method for Unsupervised Linear Discriminant Analysis

The recent work Unsupervised Linear Discriminant Analysis (Un-LDA) completes its clustering process during the alternating optimization by converting equivalently the objective and finally using the K-means algorithm. However, the K-means algorithm has its inherent drawbacks. It is hard for the K-means algorithm to deal well with some complex clustering cases where there are too many real clusters or non-convex clusters. In this paper, a novel clustering optimization method is presented to accomplish the clustering process in Un-LDA and the resulting method can be named Un-LDA(CD). Specifically, instead of the K-means algorithm, an elaborately designed coordinate descent algorithm is adopted to obtain the clusters after the objective function goes through a series of simple but deft equivalent conversions. Extensive experiments have demonstrated that the coordinate descent clustering solution for Un-LDA can outperform the original K-means based solution on the tested data sets especially those complex data sets with a pretty large number of real clusters.

Self-Weighted Unsupervised LDA.

As a hot topic in unsupervised learning, clustering methods have been greatly developed. However, the model becomes more and more complex, and the number of parameters becomes more and more with the continuous development of clustering methods. And parameter-tuning in most methods is a laborious work due to its complexity and unpredictability. How to propose a concise and beautiful model in which the parameters can be learned adaptively becomes a very meaningful problem. Aim at tackling this problem, we develop a novel self-weighted unsupervised linear discriminative analysis method, namely SWULDA. The proposed method not only avoids adjusting parameters but also explains the link between k-means and linear discriminant analysis (LDA). To obtain superior structural performance, the idea of minimizing the within-class scatter matrix and maximizing the between-class scatter matrix is embedded in the unsupervised model. Moreover, equipped with the proposed quadratic weighted optimization framework, the parameter can be adaptively learned. The extensive experiments on several datasets are conducted to validate the effectiveness of our method.

Spectral–Spatial and Superpixelwise Unsupervised Linear Discriminant Analysis for Feature Extraction and Classification of Hyperspectral Images

Spectral–Spatial and Superpixelwise Unsupervised Linear Discriminant Analysis for Feature Extraction and Classification of Hyperspectral Images

Flexible Gabor-Based Superpixel-Level Unsupervised LDA for Hyperspectral Image Classification

Hyperspectral images encompass abundant information and provide unique characteristics for material classification. However, the labeling of training samples can be challenging in hyperspectral image classification. To address this problem, this study proposes a framework named flexible Gabor-based superpixel-level unsupervised linear discriminant analysis (FG- <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Su</i> ULDA) to extract the most informative and discriminating features for classification. First, a number of 3-D flexible Gabor filters are rigorously designed using an asymmetric sinusoidal wave to sufficiently characterize the spatial–spectral structure in hyperspectral images. Then, an unsupervised linear discriminant analysis strategy guided by the entropy rate superpixel (ERS) segmentation algorithm, called <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Su</i> ULDA, is skillfully introduced to reduce the extracted large amount of FG features. The <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Su</i> ULDA method not only boosts the classification capability but also increases the peculiarity of features, with the aid of superpixel information. Finally, the achieved features are imported to the popular support vector machine classifier. The proposed FG- <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Su</i> ULDA framework is applied to four real hyperspectral image data sets, and the experiments constantly prove that our FG- <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Su</i> ULDA is superior to several state-of-the-art methods in both classification performance and computational efficiency, especially with scarce training samples. The codes of this work are available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">http://jiasen.tech/papers/</uri> for the sake of reproducibility.

Unsupervised Linear Discriminant Analysis for Jointly Clustering and Subspace Learning

Linear discriminant analysis (LDA) is one of commonly used supervised subspace learning methods. However, LDA will be powerless faced with the no-label situation. In this paper, the unsupervised LDA (Un-LDA) is proposed and first formulated as a seamlessly unified objective optimization which guarantees convergence during the iteratively alternative solving process. The objective optimization is in both the ratio trace and the trace ratio forms, forming a complete framework of a new approach to jointly clustering and unsupervised subspace learning. The extension of LDA into Un-LDA enables to not only complete unsupervised subspace learning via the explicitly presented subspace projection matrix but also simultaneously finish clustering and even clustering out-of-sample data via the explicitly presented transformation matrix. To overcome the difficulty in solving the non-convex objective optimization, we mathematically prove that the Un-LDA optimization in both forms can be transformed into the simple K-means clustering optimization when the subspace is determined. The Un-LDA optimization is eventually completed by alternatively optimizing the clusters using K-means and the subspace using the supervised LDA methods and iterating this whole process until convergence or stopping criterion. The experiments demonstrate that our proposed Un-LDA algorithms are comparable or even much superior to the counterparts.

A multivariate analytical method to characterize sediment attributes from high-frequency acoustic backscatter and ground-truthing data (Jade Bay, German North Sea coast)

One of the burning issues on the topic of acoustic seabed classification is the lack of solid, repeatable, statistical procedures that can support the verification of acoustic variability in relation to seabed properties. Acoustic sediment classification schemes often lead to biased and subjective interpretation, as they ultimately aim at an oversimplified categorization of the seabed based on conventionally defined sediment types. However, grain size variability alone cannot be accounted for acoustic diversity, which will be ultimately affected by multiple physical processes, scale of heterogeneity, instrument settings, data quality, image processing and segmentation performances. Understanding and assessing the weight of all of these factors on backscatter is a difficult task, due to the spatially limited and fragmentary knowledge of the seabed from of direct observations (e.g. grab samples, cores, videos). In particular, large-scale mapping requires an enormous availability of ground-truthing data that is often obtained from heterogeneous and multidisciplinary sources, resulting into a further chance of misclassification. Independently from all of these limitations, acoustic segments still contain signals for seabed changes that, if appropriate procedures are established, can be translated into meaningful knowledge. In this study we design a simple, repeatable method, based on multivariate procedures, with the scope to classify a 100km2, high-frequency (450kHz) sidescan sonar mosaic acquired in the year 2012 in the shallow upper-mesotidal inlet of the Jade Bay (German North Sea coast). The tool used for the automated classification of the backscatter mosaic is the QTC SWATHVIEWTMsoftware. The ground-truthing database included grab sample data from multiple sources (2009–2011). The method was designed to extrapolate quantitative descriptors for acoustic backscatter and model their spatial changes in relation to grain size distribution and morphology. The modelled relationships were used to: 1) asses the automated segmentation performance, 2) obtain a ranking of most discriminant seabed attributes responsible for acoustic diversity, 3) select the best-fit ground-truthing information to characterize each acoustic class. Using a supervised Linear Discriminant Analysis (LDA), relationships between seabed parameters and acoustic classes discrimination were modelled, and acoustic classes for each data point were predicted. The model predicted a success rate of 63.5%. An unsupervised LDA was used to model relationships between acoustic variables and clustered seabed categories with the scope of identifying misrepresentative ground-truthing data points. The model prediction scored a success rate of 50.8%. Misclassified data points were disregarded for final classification. Analyses led to clearer, more accurate appreciation of relationship patterns and improved understanding of site-specific processes affecting the acoustic signal. Value to the qualitative classification output was added by comparing the latter with a more recent set of acoustic and ground-truthing information (2014). Classification resulted in the first acoustic sediment map ever produced in the area and offered valuable knowledge for detailed sediment variability. The method proved to be a simple, repeatable strategy that may be applied to similar work and environments.

A clustering algorithm for stream data with LDA-based unsupervised localized dimension reduction

We present an algorithm for clustering high dimensional streaming data. The algorithm incorporates dimension reduction into the stream clustering framework. When a new datum arrives, the algorithm performs dimension reduction to find a local projected subspace using unsupervised LDA (Linear Discriminant Analysis)-based method. The obtained local subspace would maximally separate the nearby micro-clusters with respect to the incoming point. Then, the incoming point is assigned to a micro-cluster in the projected space, rather than in the full dimensional space. The experimental results show that the proposed algorithm outperforms its counterpart streaming clustering algorithms. Moreover, when compared with traditional clustering algorithms which require the whole data set, the proposed algorithms shows comparable clustering performances with much less computation time for large data sets.

A graph regularized dimension reduction method for out-of-sample data

Among various dimension reduction techniques, Principal Component Analysis (PCA) is specialized in treating vector data, whereas Laplacian embedding is often employed for embedding graph data. Moreover, graph regularized PCA, a combination of both techniques, has also been developed to assist the learning of a low dimensional representation of vector data by incorporating graph data. However, these approaches are confronted by the out-of-sample problem: each time when new data is added, it has to be combined with the old data before being fed into the algorithm to re-compute the eigenvectors, leading to enormous computational cost. In order to address this problem, we extend the graph regularized PCA to the graph regularized linear regression PCA (grlrPCA). grlrPCA eliminates the redundant calculation on the old data by first learning a linear function and then directly applying it to the new data for its dimension reduction. Furthermore, we derive an efficient iterative algorithm to solve grlrPCA optimization problem and show the close relatedness of grlrPCA and unsupervised Linear Discriminant Analysis at infinite regularization parameter limit. The evaluations of multiple metrics on seven realistic datasets demonstrate that grlrPCA outperforms established unsupervised dimension reduction algorithms.

Vodka on ice? Unveiling Russian media perceptions of the Arctic

This article describes Russian Arctic policy agendas as they have been reflected in mainstream Russian media outlets. The research was based on modeling topic structures of three federal and three regional newspapers. Topic modeling was performed with the unsupervised LDA algorithm and complemented with hand labelling of topics. Data was collected by retrieval of relevant newspaper articles from the media database Integrum for the period 2011–2015 (N=611); the corpus was further divided in two periods (2011–2013 and 2014–2015) to account for the potential effect of the Ukrainian crisis on agenda-setting. Both federal and regional newspapers were found to have been mostly concerned with the development of hydrocarbon resources, as coverage of this topic was the largest during both periods. However, during the second period (2014–2015), the repercussions of the Ukrainian crisis, namely the economic sanctions and securitization, gained significant attention while marginalizing other topics. The results of this analysis may serve as a foundation for application of sentiment analysis to allow an in-depth understanding of topic salience and impact on public opinion.

Unsupervised Linear Discriminant Analysis for Supporting DPGMM Clustering in the Zero Resource Scenario

In this work we make use of unsupervised linear discriminant analysis (LDA) to support acoustic unit discovery in a zero resource scenario. The idea is to automatically find a mapping of feature vectors into a subspace that is more suitable for Dirichlet process Gaussian mixture model (DPGMM) based clustering, without the need of supervision. Supervised acoustic modeling typically makes use of feature transformations such as LDA to minimize intra-class discriminability, to maximize inter-class discriminability and to extract relevant informations from high-dimensional features spanning larger contexts. The need of class labels makes it difficult to use this technique in a zero resource setting where the classes and even their amount are unknown. To overcome this issue we use a first iteration of DPGMM clustering on standard features to generate labels for the data, that serve as basis for learning a proper transformation. A second clustering operates on the transformed features. The application of unsupervised LDA demonstrably leads to better clustering results given the unsupervised data. We show that the improved input features consistently outperform our baseline input features.

Candidates for synergies: linear discriminants versus principal components.

Movement primitives or synergies have been extracted from human hand movements using several matrix factorization, dimensionality reduction, and classification methods. Principal component analysis (PCA) is widely used to obtain the first few significant eigenvectors of covariance that explain most of the variance of the data. Linear discriminant analysis (LDA) is also used as a supervised learning method to classify the hand postures corresponding to the objects grasped. Synergies obtained using PCA are principal component vectors aligned with dominant variances. On the other hand, synergies obtained using LDA are linear discriminant vectors that separate the groups of variances. In this paper, time varying kinematic synergies in the human hand grasping movements were extracted using these two diametrically opposite methods and were evaluated in reconstructing natural and American sign language (ASL) postural movements. We used an unsupervised LDA (ULDA) to extract linear discriminants. The results suggest that PCA outperformed LDA. The uniqueness, advantages, and disadvantages of each of these methods in representing high-dimensional hand movements in reduced dimensions were discussed.

Regularized Semi-Supervised Latent Dirichlet Allocation for visual concept learning

Topic model is a popular tool for visual concept learning. Most topic models are either unsupervised or fully supervised. In this paper, to take advantage of both limited labeled training images and rich unlabeled images, we propose a novel regularized Semi-Supervised Latent Dirichlet Allocation (r-SSLDA) for learning visual concept classifiers. Instead of introducing a new complex topic model, we attempt to find an efficient way to learn topic models in a semi-supervised way. Our r-SSLDA considers both semi-supervised properties and supervised topic model simultaneously in a regularization framework. Furthermore, to improve the performance of r-SSLDA, we introduce the low rank graph to the framework. Experiments on Caltech 101 and Caltech 256 have shown that r-SSLDA outperforms both unsupervised LDA and achieves competitive performance against fully supervised LDA with much fewer labeled images.

Dimensionality estimation for optimal detection of functional networks in BOLD fMRI data

Estimation of the intrinsic dimensionality of fMRI data is an important part of data analysis that helps to separate the signal of interest from noise. We have studied multiple methods of dimensionality estimation proposed in the literature and used these estimates to select a subset of principal components that was subsequently processed by linear discriminant analysis (LDA). Using simulated multivariate Gaussian data, we show that the dimensionality that optimizes signal detection (in terms of the receiver operating characteristic (ROC) metric) goes through a transition from many dimensions to a single dimension as a function of the signal-to-noise ratio. This transition happens when the loci of activation are organized into a spatial network and the variance of the networked, task-related signals is high enough for the signal to be easily detected in the data. We show that reproducibility of activation maps is a metric that captures this switch in intrinsic dimensionality. Except for reproducibility, all of the methods of dimensionality estimation we considered failed to capture this transition: optimization of Bayesian evidence, minimum description length, supervised and unsupervised LDA prediction, and Stein's unbiased risk estimator. This failure results in sub-optimal ROC performance of LDA in the presence of a spatially distributed network, and may have caused LDA to underperform in many of the reported comparisons in the literature. Using real fMRI data sets, including multi-subject group and within-subject longitudinal analysis we demonstrate the existence of these dimensionality transitions in real data.

Automated classification of multi-spectral MR images using Linear Discriminant Analysis

Magnetic resonance imaging (MRI) is a valuable instrument in medical science owing to its capabilities in soft tissue characterization and 3D visualization. A potential application of MRI in clinical practice is brain parenchyma classification. This work proposes a novel approach called “Unsupervised Linear Discriminant Analysis (ULDA)” to classify and segment the three major tissues, i.e. gray matter (GM), white matter (WM) and cerebral spinal fluid (CSF), from a multi-spectral MR image of the human brain. The ULDA comprises two processes, namely Target Generation Process (TGP) and Linear Discriminant Analysis (LDA) classification. TGP is a fuzzy-set process that generates a set of potential targets from unknown information, and applies these targets to train the optimal division boundary by LDA, such that three tissues GM, WM and CSF are separated. Finally, two sets of images, namely computer-generated phantom images and real MR images are used in the experiments to evaluate the effectiveness of ULDA. Experiment results reveal that UDLA segments a multi-spectral MR image much more effectively than either FMRIB's Automated Segmentation Tool (FAST) or Fuzzy C-means (FC).