Scatter Matrices Research Articles

Comparison of Matrix Decomposition in Null Space-Based LDA Method

Problems with small sample sizes and high dimensionality are common in pattern recognition. Almost all machine learning algorithms degrade in high-dimensional data, so that singularities in the scatter matrices, the main problem of the Linear Discriminant Analysis (LDA) technique, might result. A null space-based LDA (NLDA) has been conceived to address the singularity issue. NLDA aims to maximize the distance between classes in the null space of the within-class scatter matrix. In the first research, the NLDA method was performed by computing the eigenvalue decomposition and singular value decomposition (SVD). This research led to several new implementations of the NLDA method that use other matrix decompositions. The new implementations include NLDA using Cholesky decomposition and NLDA using QR decomposition. This paper compares the performance of three NLDA methods that use different matrix decompositions, namely, SVD, Cholesky decomposition, and QR decomposition. Two sets of data were used in the experiments that used three different NLDA algorithms. To determine the performance of the NLDA methods, the classification accuracy of the three methods was measured using the confusion matrix. The results show that the NLDA method using SVD has the best performance when compared to the other two methods, achieving a precision of 77.8% accuracy for the Colon dataset and a precision of 98.8% accuracy for the TKI-resistance dataset.

Tandem clustering with invariant coordinate selection

For multivariate data, tandem clustering is a well-known technique aiming to improve cluster identification through initial dimension reduction. Nevertheless, the usual approach using principal component analysis (PCA) has been criticized for focusing solely on inertia so that the first components do not necessarily retain the structure of interest for clustering. To address this limitation, a new tandem clustering approach based on invariant coordinate selection (ICS) is proposed. By jointly diagonalizing two scatter matrices, ICS is designed to find structure in the data while providing affine invariant components. Certain theoretical results have been previously derived and guarantee that under some elliptical mixture models, the group structure can be highlighted on a subset of the first and/or last components. However, ICS has garnered minimal attention within the context of clustering. Two challenges associated with ICS include choosing the pair of scatter matrices and selecting the components to retain. For effective clustering purposes, it is demonstrated that the best scatter pairs consist of one scatter matrix capturing the within-cluster structure and another capturing the global structure. For the former, local shape or pairwise scatters are of great interest, as is the minimum covariance determinant (MCD) estimator based on a carefully chosen subset size that is smaller than usual. The performance of ICS as a dimension reduction method is evaluated in terms of preserving the cluster structure in the data. In an extensive simulation study and empirical applications with benchmark data sets, various combinations of scatter matrices as well as component selection criteria are compared in situations with and without outliers. Overall, the new approach of tandem clustering with ICS shows promising results and clearly outperforms the PCA-based approach.

Numerically stable locality-preserving partial least squares discriminant analysis for efficient dimensionality reduction and classification of high-dimensional data

Dimensionality reduction plays a pivotal role in preparing high-dimensional data for classification and discrimination tasks by eliminating redundant features and enhancing the efficiency of classifiers. The effectiveness of a dimensionality reduction algorithm hinges on its numerical stability. When data projections are numerically stable, they lead to enhanced class separability in the lower-dimensional embedding, consequently yielding higher classification accuracy. This paper investigates the numerical attributes of dimensionality reduction and discriminant subspace learning, with a specific focus on Locality-Preserving Partial Least Squares Discriminant Analysis (LPPLS-DA). High-dimensional data frequently introduce singularity in the scatter matrices, posing a significant challenge. To tackle this issue, the paper explores two robust implementations of LPPLS-DA. These approaches not only optimize data projections but also capture more discriminative features, resulting in a marked improvement in classification accuracy. Empirical evidence supports these findings through numerical experiments conducted on synthetic and spectral datasets. The results demonstrate the superior performance of the proposed methods when compared to several state-of-the-art dimensionality reduction techniques in terms of both classification accuracy and dimension reduction.

Robust second-order stationary spatial blind source separation using generalized sign matrices

Consider a spatial blind source separation model in which the observed multivariate spatial data are assumed to be a linear mixture of latent stationary spatially uncorrelated random fields. The objective is to recover an unknown mixing procedure as well as the latent random fields. Recently, spatial blind source separation methods that are based on the simultaneous diagonalization of two or more scatter matrices were proposed. In cases involving uncontaminated data, such methods can solve the blind source separation problem, however, in the presence of outlying observations, these methods perform poorly. We propose a robust blind source separation method that employs robust global and local covariance matrices based on generalized spatial signs in simultaneous diagonalization. Simulation studies are employed to illustrate the robustness and efficiency of the proposed methods in various scenarios.

Robust signal dimension estimation via SURE

The estimation of signal dimension under heavy-tailed latent variable models is studied. As a primary contribution, robust extensions of an earlier estimator based on Gaussian Stein’s unbiased risk estimation are proposed. These novel extensions are based on the framework of elliptical distributions and robust scatter matrices. Extensive simulation studies are conducted in order to compare the novel methods with several well-known competitors in both estimation accuracy and computational speed. The novel methods are applied to a financial asset return data set.

On testing the equality of latent roots of scatter matrices under ellipticity

In the present paper, we tackle the problem of testing H0q:λq>λq+1=⋯=λp, where λ1,…,λp are the scatter matrix eigenvalues of an elliptical distribution on Rp. This is a classical problem in multivariate analysis which is very useful in dimension reduction. We analyse the problem using the Le Cam asymptotic theory of experiments and show that contrary to the testing problems on eigenvalues and eigenvectors of a scatter matrix tackled in Hallin et al. (2010), the non-specification of nuisance parameters has an asymptotic cost for testing H0q. We moreover derive signed-rank tests for the problem that enjoy the property of being asymptotically distribution-free under ellipticity. The van der Waerden rank test uniformly dominates the classical pseudo-Gaussian procedure for the problem. Numerical illustrations show the nice finite-sample properties of our tests.

Numerical Considerations and a new implementation for invariant coordinate selection

Invariant coordinate selection (ICS) is a multivariate data transformation and a dimension reduction method that can be useful in many different contexts. It can be used for outlier detection or cluster identification, and can be seen as an independent component or a non-Gaussian component analysis method. The usual implementation of ICS is based on a joint diagonalization of two scatter matrices, and may be numerically unstable in some ill-conditioned situations. We focus on one-step M-scatter matrices and propose a new implementation of ICS based on a pivoted QR factorization of the centered data set. This factorization avoids the direct computation of the scatter matrices and their inverse and brings numerical stability to the algorithm. Furthermore, the row and column pivoting leads to a rank revealing procedure that allows computation of ICS when the scatter matrices are not full rank. Several artificial and real data sets illustrate the interest of using the new implementation compared to the original one.

Hybrid Riemannian Graph-Embedding Metric Learning for Image Set Classification

With the continuously increasing amount of video data, image set classification has recently received widespread attention in the CV&PR community. However, the intra-class diversity and inter-class ambiguity of representations remain an open challenge. To tackle this issue, several methods have been put forward to perform multiple geometry-aware image set modelling and learning. Although the extracted complementary geometric information is beneficial for decision making, the sophisticated computational paradigm (e.g., scatter matrices computation and iterative optimisation) of such algorithms is counterproductive. As a countermeasure, we propose an effective hybrid Riemannian metric learning framework in this paper. Specifically, we design a multiple graph embedding-guided metric learning framework for the sake of fusing these complementary kernel features, obtained via the explicit RKHS embeddings of the Grassmannian manifold, SPD manifold, and Gaussian embedded Riemannian manifold, into a unified subspace for classification. Furthermore, the involved optimisation problem of the developed model can be solved in terms of a series of sub-problems, achieving improved efficiency theoretically and experimentally. Substantial experiments are carried out to evaluate the efficacy of our approach. The experimental results suggest the superiority of it over the state-of-the-art methods.

On the distribution of sample scale-free scatter matrices

On the distribution of sample scale-free scatter matrices

Saliency-Based Multilabel Linear Discriminant Analysis.

Linear discriminant analysis (LDA) is a classical statistical machine-learning method, which aims to find a linear data transformation increasing class discrimination in an optimal discriminant subspace. Traditional LDA sets assumptions related to the Gaussian class distributions and single-label data annotations. In this article, we propose a new variant of LDA to be used in multilabel classification tasks for dimensionality reduction on original data to enhance the subsequent performance of any multilabel classifier. A probabilistic class saliency estimation approach is introduced for computing saliency-based weights for all instances. We use the weights to redefine the between-class and within-class scatter matrices needed for calculating the projection matrix. We formulate six different variants of the proposed saliency-based multilabel LDA (SMLDA) based on different prior information on the importance of each instance for their class(es) extracted from labels and features. Our experiments show that the proposed SMLDA leads to performance improvements in various multilabel classification problems compared to several competing dimensionality reduction methods.

Robust approaches to redundancy analysis

Redundancy analysis is used to examine the interrelationships between two sets of variables, one set being dependent on the other. It aims to maximize the proportion of variance of the dependent variables that can be explained by each successive uncorrelated linear combination of the explanatory variables. It is an alternative to canonical correlation analysis when there is no symmetry in the variables. In this work, we study two different approaches to robustly estimate redundancy parameters. As a first approach, we consider a plug-in method based on robust correlation matrices capable of achieving simultaneously high efficiency under a Gaussian model and high resistance to outliers. As a second approach, we explore the relationship between redundancy analysis and multivariate linear regression and propose a method based on robust multivariate linear regression estimators. For elliptical distributions, the local robustness of the redundancy analysis based on robust scatter matrices is studied using the influence function. A simulation study shows that robust estimators perform better than the classical estimator and compares the proposals under contaminated samples. The performance of the proposals in a real data example is presented.

Application of integrated wave-to-wire modelling for the preliminary design of oscillating water column systems for installations in moderate wave climates

Oscillating water column (OWC) devices are the technical solution more favourably considered for a significant future exploitation of the energy potential of sea waves. While theoretically quite simple, these systems are indeed rather complex in terms of correct tuning between the caisson and turbine. Recent studies showed that only a holistic approach, also including turbine control, is the way of effectively performing their precise design.To this end, the research presents the application of an analytical wave-to-wire model of an OWC device for the preliminary optimisation of these systems for a small installation in a moderate wave climate in the Mediterranean Sea. More specifically, scatter matrices were determined for imposing the wave states of a selected location positioned in Tuscany (Italy). Regarding previous similar studies, a novel joint optimisation of the caisson and controlled turbine is proposed for the specific site. Either monoplane isolated Wells turbines and axial impulse turbines were analysed. The operating curves, performance parameters, and attended annual energy harvesting were calculated, providing an interesting overview of the functioning of the system. The optimised devices with the Wells and impulse turbomachines convert 36.71 and 42.17 MWh/year operating with a global efficiency of 6.82% and 7.84%, respectively.

Multiparty Data Publishing via Blockchain and Differential Privacy

Data are distributed between different parties. Collecting data from multiple parties for analysis and mining will serve people better. However, it also brings unprecedented privacy threats to the participants. Therefore, safe and reliable data publishing among multiple data owners is an urgent problem to be solved. We mainly study the problem of privacy protection in data publishing. For a centralized scenario, we propose the LDA-DP algorithm. First, the within-class mean vectors and the pooled within-class scatter matrix are perturbed by the Gaussian noise. Second, the optimal projection direction vector with differential privacy is obtained by the Fisher criterion. Finally, the low-dimensional projection data of the original data are obtained. For distributed scenarios, we propose the Mul-LDA-DP algorithm based on a blockchain and differential privacy technology. First, the within-class mean vectors and within-class scatter matrices of local data are perturbed by the Gaussian noise and uploaded to the blockchain network. Second, the projection direction vector is calculated in the blockchain network and returned to the data owner. Finally, the data owner uses the projection direction vector to generate low-dimensional projection data of the original data and upload it to the blockchain network for publishing. Furthermore, in a distributed scenario, we propose a correlated noise generation scheme that uses the additivity of the Gaussian distribution to mitigate the effects of noise and can achieve the same noise level as the centralized scenario. We measure the utility of the published data by the SVM misclassification rate. We conduct comparative experiments with similar algorithms on different real data sets. The experimental results show that the data released by the two algorithms can maintain good utility in SVM classification.

Data Mining and Visualization of High-Dimensional ICME Data for Additive Manufacturing

Integrated computational materials engineering (ICME) methods combining CALPHAD with process-based simulations can produce rich, high-dimensional data for alloy and process design. In ICME methods for metallurgical applications, the visualization and interpretation of such high-dimensional data has previously been through heat maps represented in 2 or 3 dimensions. While such an approach is ideal when one variable is varied at a time, in the case of high-dimensional data with multiple variables varied simultaneously, as is the case in additive manufacturing, interpreting the trends through two- or three-dimensional heat maps becomes challenging. Here, we propose a strategy of mixed visual data mining and quantitative analysis for high-dimensional metallurgical and process data using high-throughput thermodynamic calculations. Two case studies show the application of the proposed approach. The first case study investigated the effects of feedstock chemistry on the $$\delta$$ ferrite formation in 316L stainless steel powders used for binder jet additive manufacturing. The second case study linked Scheil–Gulliver calculations to a process model for dissimilar joining of aluminum alloys 5356 and 6111 during laser hot-wire additive manufacturing. Both cases contained thousands of calculated data points, showcasing the utility of visual data analysis through parallel coordinate plotting, Pearson correlation coefficient matrices, and scatter matrices compared to traditional process maps. These visualization techniques can be extended to many additive manufacturing problems to capture process–structure–property relationships for additively manufactured components.

Chemical weathering of sediment (CWS): A web-based application for chemical weathering study of rocks and sediment

CWS is an open-source application developed for calculating chemical weathering indices using major elemental geochemical data of clastic sediments and for generating associated plots. Chemical weathering indices include Chemical Index of Weathering (CIW), Chemical Index of Weathering without CaO (CIW′; expressed as CIW*), Chemical Proxy of Alteration (CPA), Chemical Index of Alteration (CIA), Plagioclase Index of Alteration (PIA), Modified Chemical Index of Alteration (CIX), Index of Compositional Variability (ICV) and Weathering Index of Parker (WIP). Also, hierarchical charts, histogram, boxplots, scatter matrices, correlation matrices, heatmap with related data tables are included along with the statistical measures of weathering indices and their correlation with Al2O3/TiO2. Additionally, it generates plots of input data used to quantify chemical weathering of clastic sediments in various compositional space diagrams, such as A - CN - K, A - CNK - FM and M - F - W compositional space diagram. Using CWS, one can do all the calculations and artwork associated with the finished report without having to use Microsoft Excel, CorelDraw or similar software. Input data source files and output calculations results data files are both in CSV format (Comma Separated Value). HTML files can be exported for plots generated by CWS. As an example of the capability of CWS to evaluate chemical weathering, we evaluated Surma and Barail sandstones from the Mizoram Foreland Basin in North-eastern India, Manasbal Lake sediments from Kashmir; channel, suspended and overbank sediments of the Peninsular rivers of Ganga basin.

Superpixel-Level Hybrid Discriminant Analysis for Hyperspectral Image Feature Extraction

For hyperspectral image (HSI) classification, it is an challenging problem to learn highly discriminative features, since the complex local/non-local spatial-spectral association is difficult to be accurately characterized. Focused on this issue, a novel superpixel-level hybrid discriminant analysis (SHDA) method is proposed, in this paper. Here, the SHDA method takes advantage of superpixel’s merit in characterizing spatial-spectral shape-adaptive structure and the powerful capability of discriminant analysis in enhancing class-separability to learn the feature representation. Moreover, the local/non-local spatial-spectral correlation information among/between superpixels is effectively excavated and fused in one framework to further improve the classification performance of features. This is achieved by first designing two specific discriminant analysis modules, i.e., superpixel-level local discriminant analysis (SLDA) and superpixel-level non-local discriminant analysis (SNDA). In the SLDA, adaptively weighted scatter matrices are defined to characterize the local spectral similarity within each superpixel and the discrepancy among adjacent superpixels. In the SNDA, superpixel-level graphs are built for capturing the non-local contextual information, where the weights of graphs are estimated based on the most similar and dissimilar superpixels. Then, the SLDA and the SNDA are effectively fused to construct the total intra/inter-superpixel scatter matrices. Finally, a joint projection transformation is obtained via solving a simple generalized eigenvalue problem. By this way, the HSI data can be projected from a high-dimensional data space into a low-dimensional feature space, where different classes of land-covers can be more accurately distinguished. Experimental results on three real hyperspectral data sets indicate that the proposed SHDA method outperforms several state-of-the-art techniques.

A new fuzzy and Gaussian distribution induced two-directional inverse FDA for feature extraction and face recognition

In the area of face recognition research, the high dimensionality of the data is indeed a crucial problem. This paper proposes a new fuzzy and Gaussian distribution induced two-directional inverse Fisher's discriminant analysis (FGD-2DIFDA) which computes the fuzzy and Gaussian distribution membership values and thereby combined the values with the training samples to obtain the class-wise mean and the global mean. These fuzzy and Gaussian function induced membership values are incorporated in inter- and intra-class scatter matrices for diminishing the effect of uncertainty formed by variation of light facial expression and pose. Finally, the eigenvalue problems of the inverse Fisher's criteria are solved to finds the optimal inverse projection vectors. The uncertainty associated in the image region is efficiently managed by the FGD-2DIFDA method than fuzzy generalised 2DFLD (FG-2DFLD) method as two membership values are employed in the former case while the later one uses only one membership value to handle such situation. The FGD-2DIFDA method has been evaluated on the AT&T (formally known as ORL), UMIST and FERET face databases using support vector machine (SVM) as classifier. Simulation results demonstrate that the proposed FGD-2DIFDA method can obtain higher recognition rate than some state-of-the-art face recognition methods.

Neighborhood linear discriminant analysis

Linear Discriminant Analysis (LDA) assumes that all samples from the same class are independently and identically distributed (i.i.d.). LDA may fail in the cases where the assumption does not hold. Particularly when a class contains several clusters (or subclasses), LDA cannot correctly depict the internal structure as the scatter matrices that LDA relies on are defined at the class level. In order to mitigate the problem, this paper proposes a neighborhood linear discriminant analysis (nLDA) in which the scatter matrices are defined on a neighborhood consisting of reverse nearest neighbors. Thus, the new discriminator does not need an i.i.d. assumption. In addition, the neighborhood can be naturally regarded as the smallest subclass, for which it is easier to be obtained than subclass without resorting to any clustering algorithms. The projected directions are sought to make sure that the within-neighborhood scatter as small as possible and the between-neighborhood scatter as large as possible, simultaneously. The experimental results show that nLDA performs significantly better than previous discriminators, such as LDA, LFDA, ccLDA, LM-NNDA, and l2,1-RLDA.

On the usage of joint diagonalization in multivariate statistics

Scatter matrices generalize the covariance matrix and are useful in many multivariate data analysis methods, including well-known principal component analysis (PCA), which is based on the diagonalization of the covariance matrix. The simultaneous diagonalization of two or more scatter matrices goes beyond PCA and is used more and more often. In this paper, we offer an overview of many methods that are based on a joint diagonalization. These methods range from the unsupervised context with invariant coordinate selection and blind source separation, which includes independent component analysis, to the supervised context with discriminant analysis and sliced inverse regression. They also encompass methods that handle dependent data such as time series or spatial data.

Asymptotic and bootstrap tests for subspace dimension

Many linear dimension reduction methods proposed in the literature can be formulated using an appropriate pair of scatter matrices. The eigen-decomposition of one scatter matrix with respect to another is then often used to determine the dimension of the signal subspace and to separate signal and noise parts of the data. Three popular dimension reduction methods, namely principal component analysis (PCA), fourth order blind identification (FOBI) and sliced inverse regression (SIR) are considered in detail and the first two moments of subsets of the eigenvalues are used to test for the dimension of the signal space. The limiting null distributions of the test statistics are discussed and novel bootstrap strategies are suggested for the small sample cases. In all three cases, consistent test-based estimates of the signal subspace dimension are introduced as well. The asymptotic and bootstrap tests are illustrated in real data examples.