Projection Matrices Research Articles

Strong convergence of the empirical spectral distribution of a unified matrix model

Let [Formula: see text] is a positive definite matrix and [Formula: see text] with [Formula: see text] being independent and identically distributed (i.i.d.) centered real random variables. Consider the matrix [Formula: see text] where [Formula: see text] and [Formula: see text] are two projection matrices (deterministic or random) satisfying [Formula: see text] and [Formula: see text]. Additionally, if [Formula: see text] and [Formula: see text] are random matrices, they are independent of [Formula: see text]. In this paper, we demonstrate that the empirical spectral distribution of [Formula: see text] converges almost surely to a non-random distribution when [Formula: see text] and [Formula: see text], assuming [Formula: see text] has finite second moment.

New Statistical Residuals for Regression Models in the Exponential Family: Characterization, Simulation, Computation, and Applications

Residuals are essential in regression analysis for evaluating model adequacy, validating assumptions, and detecting outliers or influential data. While traditional residuals perform well in linear regression, they face limitations in exponential family models, such as those based on the binomial and Poisson distributions, due to heteroscedasticity and dependence among observations. This article introduces a novel standardized combined residual for linear and nonlinear regression models within the exponential family. By integrating information from both the mean and dispersion sub-models, the new residual provides a unified diagnostic tool that enhances computational efficiency and eliminates the need for projection matrices. Simulation studies and real-world applications demonstrate its advantages in efficiency and interpretability over traditional residuals.

FedTP: Federated Learning by Transformer Personalization.

Federated learning is an emerging learning paradigm where multiple clients collaboratively train a machine learning model in a privacy-preserving manner. Personalized federated learning extends this paradigm to overcome heterogeneity across clients by learning personalized models. Recently, there have been some initial attempts to apply transformers to federated learning. However, the impacts of federated learning algorithms on self-attention have not yet been studied. In this article, we investigate this relationship and reveal that federated averaging (FedAvg) algorithms actually have a negative impact on self-attention in cases of data heterogeneity, which limits the capabilities of the transformer model in federated learning settings. To address this issue, we propose FedTP, a novel transformer-based federated learning framework that learns personalized self-attention for each client while aggregating the other parameters among the clients. Instead of using a vanilla personalization mechanism that maintains personalized self-attention layers of each client locally, we develop a learn-to-personalize mechanism to further encourage the cooperation among clients and to increase the scalability and generalization of FedTP. Specifically, we achieve this by learning a hypernetwork on the server that outputs the personalized projection matrices of self-attention layers to generate clientwise queries, keys, and values. Furthermore, we present the generalization bound for FedTP with the learn-to-personalize mechanism. Extensive experiments verify that FedTP with the learn-to-personalize mechanism yields state-of-the-art performance in the non-IID scenarios. Our code is available online https://github.com/zhyczy/FedTP.

Overview of Structure from Motion

In this paper is given an overview of tools and techniques for obtain information about the geometry of 3D scenes from 2D images. The generating three-dimensional structure from a series of 2D images or video of scenes is known as Structure from Motion (SfM). A central tenet of structure from motion is that, given the position of a feature in one image, it is possible to find the corresponding position of the same feature in successive images. We described methods for the simultaneous recovery of 3D points and camera projection matrices using corresponding image points in multiple views. Structure from Motion (SfM) is a fascinating field within computer vision that seeks to reconstruct a three-dimensional structure of the environment from a sequence of two-dimensional images. At the heart of SfM lies a set of sophisticated algorithms that enable the extraction of spatial information from a series of images. Two public repositories that may be of use are Open SfM and Colmap.

An electronic nose drift compensation algorithm based on semi-supervised adversarial domain adaptive convolutional neural network

Sensor drift is a significant challenge leading to performance degradation in electronic nose(E-nose) systems. Effectively addressing sensor drift represents the most daunting problem in E-nose technology. This work proposes a domain adaptive approach called Semi-Supervised Adversarial Domain Adaptive Convolutional Neural Network (SAD-CNN) to tackle the long-term drift in E-nose and device displacement. SAD-CNN leverages adversarial learning to minimize distribution disparities between the source and target domains. Unlike traditional methods employing projection matrices, SAD-CNN utilizes one-dimensional convolutional neural networks as feature extractors, circumventing the complexities associated with parameter adjustments and matrix calculations in the projection process. During the training process, utilizing pseudo-labels generated by the semi-supervised self-training method to train the model, thereby reducing the labeling costs. Additionally, confidence threshold screening is introduced during the self-training phase to minimize erroneous pseudo-labels. Furthermore, the regenerative Hilbert space’s Maximum Mean Difference combined with Minimum Variance is introduced as a domain constraint function to mitigate distribution discrepancies between domains and enhance feature discriminability across domains. The experimental results demonstrate that the SAD-CNN method outperforms others. Within the long-term drift dataset, the classification accuracies for different scenarios are 78.01 % and 82.53 %, respectively. Meanwhile, the instrument change dataset yields a classification accuracy of 96.45 %.

Variety is the spice of life: nongenetic variation in life histories influences population growth and evolvability.

Individual vital rates are key determinants of lifetime reproductive success, and variability in these rates shapes population dynamics. Previous studies have found that this vital rate hetero- geneity can influence demographic properties including population growth rates, however, the explicit effects of the amount of variation within and the covariance between vital rates that can also vary throughout the lifespan on population growth remains unknown. Here, we explore the analytical consequences of nongenetic heterogeneity on long-term population growth rates and rates of evolution by modifying traditional age-structured population projection matrices to incorporate variation among individual vital rates. The model allows vital rates to be permanent throughout life ("fixed condition") or to change over the lifespan ("dynamic condition"). We reduce the complexity associated with adding individual heterogeneity to age-structured models through a novel application of matrix collapsing ("phenotypic collapsing"), showing how to col- lapse in a manner that preserves the asymptotic and transient dynamics of the original matrix. The main conclusion is that nongenetic individual heterogeneity can strongly impact the long-term growth rate and rates of evolution. The magnitude and sign of this impact depends heavily on how the heterogeneity covaries across the lifespan of an organism. Our results emphasize that nongenetic variation cannot simply be viewed as random noise, but rather that it has consistent, predictable effects on fitness and evolvability.

Flexible Dual Multi-Modal Hashing for Incomplete Multi-Modal Retrieval

Multimodal hashing aims to efficiently integrate multi-source data into a unified discrete Hamming space, facilitating fast similarity searches with minimal query and storage overhead. Traditional multimodal hashing assumes that data from different sources are fully observed, an assumption that fails in real-world scenarios involving large-scale multimodal data, thereby compromising conventional methods. To address these limitations during both training and retrieval, our approach manages dual-stage data missing, occurring in both phases. In this paper, we introduce a novel framework called Flexible Dual Multimodal Hashing (FDMH), which recovers missing data at both stages by jointly leveraging low-dimensional data relations and semantic graph structural relationships in multi-source data, achieving promising performance in incomplete multimodal retrieval. We transform the original features into anchor graphs and use existing modalities to reconstruct the anchor graphs of missing modalities. Based on these anchor graphs, we perform weight-adaptive fusion in the semantic space, supervised by original semantic labels, and apply a tensor nuclear norm to enforce consistency constraints on the projection matrices across different modalities. Furthermore, our method flexibly fuses existing and recovered modalities during retrieval. We validate the effectiveness of our approach through extensive experiments on four large-scale multimodal datasets, demonstrating its robust performance in real-world dual-missing retrieval scenarios.

Cross-domain manifold structure preservation for transferable and cross-machine fault diagnosis

To address the decline or failure in the autonomous learning capability of traditional transfer learning methods when training and test samples come from different machines, resulting in low cross-machine fault diagnosis rates, we propose a cross-domain manifold structure preservation (CDMSP) method for diagnosing rolling bearing faults across machines. The CDMSP method can induce the manifold space projection matrices of the source and target domains more effectively. This method maps high-dimensional features into a low-dimensional manifold, preserving non-linear relationships and aligning distribution differences while maintaining cross-domain manifold structure consistency. Additionally, highly confidently labeled target domain samples are selected from each mapping result and added to the training dataset to enhance subspace learning in subsequent iterations. The CDMSP method is both simple and effective at capturing the underlying structures and patterns in the data. The CWRU dataset and our self-built test platform dataset were used to validate this method. Experimental results show that CDMSP, as a non-deep domain adaptation method of transfer learning, outperforms similar methods in cross-machine fault identification, achieving a maximum fault identification accuracy of 100 % with excellent convergence performance. Furthermore, simulated diagnostic experiments under noise interference indicate that CDMSP maintains high fault identification accuracy, even in noisy environments. Overall, CDMSP is an efficient and reliable new method for diagnosing cross-machine bearing faults.

Exploring corpus-invariant emotional acoustic feature for cross-corpus speech emotion recognition

Unsupervised cross-corpus speech emotion recognition (SER) is the task where the labeled training (source) and unlabeled testing (target) speech come from different corpora. Subspace transfer learning is one of the mainstream technologies for tackling cross-corpus SER challenges, which is achieved by utilizing projection matrices to learn common corpus-invariant feature representations between the source and target corpus. However, these methods mainly focus on mapping modeling from the input low-level descriptor (LLD) feature space to the corpus-invariant emotional feature space, which is an implicit feature mapping process that lacks interpretability for the selected features. This omission leads to an inability to pinpoint which acoustic features possess corpus invariance. To bridge this gap, we first propose a new transfer subspace learning framework with feature selection capabilities, i.e., the Corpus-Invariant Emotional Acoustic Feature Seeker (CAFS). Specifically, the CAFS integrates two core terms into the transfer regression loss function: (1) the emotion preservation term: This term includes emotional regression and the l2,1 norm, which is mainly used to select features and ensure that these features are related to emotions. (2) the corpus invariance preservation term: This item is mainly used to measure the difference in feature distribution between the source and target corpora. Minimizing this term bridges the gap between the source and target domains, ensuring that the chosen acoustic features are corpus-invariant. Subsequently, we conducted extensive cross-corpus SER experiments to explore corpus-invariant emotional acoustic features under various commonly used acoustic feature sets (IS09 and eGeMAPS). Through statistical analysis of the acoustic features sought by the CAFS framework, some acoustic features (e.g., Mel-Frequency Cepstral Coefficients (MFCC) and Formant) reveal their corpus-invariant properties, which could provide insights for feature selection in cross-corpus SER. These findings also lay the groundwork for a solid theoretical and empirical foundation for future research and applications in cross-corpus SER.

Source number estimation based on sequential ratio for the invariant subspace of matrices

In this paper, we propose a method to estimate the number of sources based on the sequential ratio for the invariant subspace of matrices that can provide accurate results in scenarios involving colored noise and a small number of snapshots. We first form a pair of projection matrices on each eigenvector of the sample covariance matrices by using the invariant subspace of matrices. We then use the generalized cosine-squared method to evaluate the differences of M pairs of projection matrices, according to the gap between the projection matrices corresponding to the eigenvectors of the signals and noise. This yields a distinct transition from the signal subspace to the noise subspace. Based on this, we develop a sequential ratio-based method to precisely determine the number of sources by identifying the position at which the ratio of differences significantly decreases. The proposed method can identify the index value at which a significant drop occurs in the number of sources according to the transition from the signal subspace to the noise subspace. The simulation results showed that the proposed method is superior to the current methods, and excels particularly in conditions involving colored noise.

Fine-grained personalized federated learning via transformer in the transformer framework

Federated learning (FL) stands out as one of the most decentralized and privacy-preserving machine learning techniques. FL enables collaborative model training across multiple clients while preserving the privacy of their data. An extension of this method, namely, personalized federated learning (PFL), seeks to address client data heterogeneity by tailoring individualized models. Existing PFL methods that use transformers do not always take into account how federated averaging algorithms (a common FL algorithm that takes a weighted average of model parameters) can aggregate the self-attention mechanism. This makes it harder for transformers to work in federated settings. In addition, these approaches typically underperform on fine-grained tasks. To overcome these limitations, this study develops a novel transformer-in-transformer facilitated FL framework that not only aggregates global model parameters but also pinpoints personalized parameters for each client. This framework departs from traditional methods by implementing an “adaptive-personalization” mechanism, which not only enhances inter-client cooperation but also improves the model's scalability and adaptability. On the server side, a hypernetwork is developed to generate personalized projection matrices for use in the self-attention layers. These matrices determine the query, key, and value parameters for each client. Experimental results demonstrate that the model equipped with this adaptive-personalization mechanism considerably outperforms traditional models in scenarios with heterogeneous data, thus validating its effectiveness and efficiency.

Testing for independence of sets of high-dimensional normal vectors using random projection approach

A simple test is proposed to test the independence of high-dimensional random normal vectors. The method consists of two steps. First, the primary high-dimensional data is projected onto a low-dimensional subspace multiple times using random projection matrices. Second, the test statistic is constructed by utilizing the classical statistics obtained from the projected low-dimensional datasets. Simulations are performed to compare the performance of the proposed test with existing state-of-the-art tests, in terms of test sizes and powers. Finally, the proposed methodology is illustrated using two gene datasets, namely the Colon and Leukemia cancer datasets.

Off-diagonal elements of projection matrices and dimension asymptotics

We provide insights on asymptotic behavior of the off-diagonal elements of projection matrices in settings, where the dimension of underlying vectors grows with the sample size. Under designs favorable to application of the random matrix theory, the off-diagonal elements are asymptotically normal with a simple variance expression. We also discuss the robustness of the result to deviations of the design from the ideal setup.

Eigenspectrum regularisation reverse neighbourhood discriminative learning

AbstractLinear discriminant analysis is a classical method for solving problems of dimensional reduction and pattern classification. Although it has been extensively developed, however, it still suffers from various common problems, such as the Small Sample Size (SSS) and the multimodal problem. Neighbourhood linear discriminant analysis (nLDA) was recently proposed to solve the problem of multimodal class caused by the contravention of independently and identically distributed samples. However, due to the existence of many small‐scale practical applications, nLDA still has to face the SSS problem, which leads to instability and poor generalisation caused by the singularity of the within‐neighbourhood scatter matrix. The authors exploit the eigenspectrum regularisation techniques to circumvent the singularity of the within‐neighbourhood scatter matrix of nLDA, which is called Eigenspectrum Regularisation Reverse Neighbourhood Discriminative Learning (ERRNDL). The algorithm of nLDA is reformulated as a framework by searching two projection matrices. Three eigenspectrum regularisation models are introduced to our framework to evaluate the performance. Experiments are conducted on the University of California, Irvine machine learning repository and six image classification datasets. The proposed ERRNDL‐based methods achieve considerable performance.

Identification of dominant subspaces for model reduction of structured parametric systems

SummaryIn this paper, we discuss a novel model reduction framework for linear structured dynamical systems. The transfer functions of these systems are assumed to have a special structure, for example, coming from second‐order linear systems or time‐delay systems, and they may also have parameter dependencies. Firstly, we investigate the connection between classic interpolation‐based model reduction methods with the reachability and observability subspaces of linear structured parametric systems. We show that if enough interpolation points are taken, the projection matrices of interpolation‐based model reduction encode these subspaces. Consequently, we are able to identify the dominant reachable and observable subspaces of the underlying system. Based on this, we propose a new model reduction algorithm combining these features and leading to reduced‐order systems. Furthermore, we discuss computational aspects of the approach and its applicability to a large‐scale setting. We illustrate the efficiency of the proposed approach with several numerical large‐scale benchmark examples.

Model order reduction for discrete time-delay systems based on Laguerre function expansion

We investigate model order reduction for discrete time-delay systems via orthogonal polynomial expansion in the frequency domain. The transfer function of systems is expanded in the framework of Laguerre function basis. We show that Laguerre coefficients of the states satisfy a linear system and reduced models generated by projection methods can preserve some Laguerre coefficients of the original systems. We prove that the subspace spanned by Laguerre coefficients is exactly a high order Krylov subspace, thereby leading to an efficient computation of projection matrices and a more accurate coefficient-matching property in the two-sided framework. Further the Laguerre expansion of systems is employed to enable an approximate but fast execution of balanced truncation for discrete time-delay systems. Specifically, Gramians are approximated based on the derived Laguerre coefficients, which circumvents the main bottleneck of balanced truncation methods. Finally, two numerical examples are simulated to demonstrate the feasibility and effectiveness of the proposed methods.

Abstract 4907: RanBALL: Identifying B-cell acute lymphoblastic leukemia subtypes based on an ensemble random projection model

Abstract As the most common pediatric malignancy, B-cell acute lymphoblastic leukemia (B-ALL) has multiple distinct subtypes characterized by recurrent and sporadic somatic and germline genetic alterations like chromosomal alteration, transcription factor rearrangement or kinase inhibition. The treatment of B-ALL patients is personalized based on specific subtypes, as the treatment responses for different B-ALL subtypes may vary considerably. Identification of B-ALL subtypes can facilitate risk stratification and enable tailored therapeutic approaches. Existing methods for B-ALL subtyping primarily depend on immunophenotypic, cytogenetic and genomic analyses, which would be costly, complicated, and laborious in clinical practice applications. To overcome these challenges, we present RanBALL (an Ensemble Random Projection-Based Model for Identifying B-Cell Acute Lymphoblastic Leukemia Subtypes), an accurate and cost-effective model for B-ALL subtype identification based on transcriptomic profiling only. RanBALL leverages random projection (RP) to construct an ensemble of dimension-reduced multi-class classifiers for B-ALL subtyping. Specifically, the transcriptomic profiling features were projected onto low-dimensional spaces by random projection matrices whose elements conform to a distribution characterized by zero mean and unit variance. To ensure reliable and robust performance, we selected 20 subspace dimensions ranging from 600 to 2500, with intervals of 100. The transformed low dimensional data matrix was used for training an ensemble of multi-class support vector machine (SVM) classifiers, each corresponding to one of the RP matrices of various dimensions. The predicted probabilistic scores of each B-ALL subtype were integrated for determining the final decision. Results based on 10 times 10-fold cross validation tests for >1700 B-ALL patients demonstrated that the proposed model achieved an accuracy of 93.7%, indicating promising prediction capabilities of RanBALL for B-ALL subtyping. Furthermore, the 30% held-out tests suggested that the model was robust and consistent to maintain high confidence levels for accurate predictions. The high accuracies of RanBALL suggested that our model could effectively capture underlying patterns of transcriptomic profiling for accurate B-ALL subtype identification. To extend the impact of RanBALL, we have established a free and publicly available python package for RanBALL available at https://github.com/wan-mlab/RanBALL. We believe RanBALL will facilitate the discovery of B-ALL subtype-specific marker genes and therapeutic targets, and eventually have consequential positive impacts on downstream risk stratification and tailored treatment design. Citation Format: Lusheng Li, Hanyu Xiao, Joseph D. Khoury, Jieqiong Wang, Shibiao Wan. RanBALL: Identifying B-cell acute lymphoblastic leukemia subtypes based on an ensemble random projection model [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2024; Part 1 (Regular Abstracts); 2024 Apr 5-10; San Diego, CA. Philadelphia (PA): AACR; Cancer Res 2024;84(6_Suppl):Abstract nr 4907.

Balancing: A solution to our distorted indices of transient dynamics

AbstractBecause populations are at the mercy of random disturbances large and small, they rarely, if ever, converge on predicted long‐term behaviors. Therefore, when employing matrix population models, ecologists study the dynamics of populations that depart from stable distributions. Necessary for such studies are indices of transient dynamics that measure the size of short‐term population fluctuations. These indices advance our understanding of population dynamics by revealing that population growth rate in a single timestep can far exceed the stable population growth rate. Despite their value, indices of transient behavior possess two major shortcomings: they are scale dependent and easily distorted by outsized population classes. Distortion occurs whenever immature classes, due to their sheer size, carry greater weight in the calculation of population size than mature classes. Beluga sturgeon (Huso huso), for example, have an immature age class (eggs) that is several orders of magnitude larger than its mature age classes. To remove the undue influence of outsized classes, I use balancing, which rescales classes by the stable population distribution and makes the indices of transient dynamics scale invariant. I apply balancing to 1800 population projection matrices for various species across the Animal Kingdom, using reactivity and the Henrici metric of non‐normality as indices of transient dynamics. I found that balancing profoundly changes the picture of which populations have the greatest or least potential transient dynamics. Using a population projection matrix for northern pike (Esox lucius), I demonstrate how balancing influences pseudospectra contour plots that are used to infer transient dynamics.

Assessing built microclimate with building group frontal projection maps: A sun-path-dependent deep transfer learning neural network approach

Climate conditions play a pivotal role in estimating buildings' energy consumption. For urban-scale building energy dynamic simulation, the typical meteorological year weather represents a universal climate condition for the entire city. However, the varying terrain roughness, environment conditions, human-built architectures jointly formulate a unique microclimate condition for each building. This study intends to develop flexible and reliable approach to assess a building's microclimate condition based on its surrounding environment morphological features. The proposed approach leverages deep transfer learning neural networks that combines seasonal sun path trajectories and front projection maps of building groups. To validate the proposed method, a case study was conducted on a campus environment with wireless environmental sensing systems. The findings of this study demonstrate that the projection matrices and sun path can improve the assessment methods solely based on the dynamic impacts of urban morphological features and can forecast microclimate dynamics for urban-scale building energy simulation.

Locality Cross-domain Discriminant Analysis for Membranous Nephropathy Recognition Using Microscopic Hyperspectral Imaging.

Cross-domain methods have been proposed to learn the domain invariant knowledge that can be transferred from the source domain to the target domain. Existing cross-domain methods attempt to minimize the distribution discrepancy of the domains. However, these methods fail to explore the domain invariant subspace due to the samples of different classes between two domains may overlap in the new subspace. They consider the features in the original space data that may be unnecessary or irrelevant to the final classification, and neglect to preserve the local manifold structure between two domains. To solve these problems, a novel feature extraction method called Locality Cross-domain Discriminant Analysis (LCDA) is proposed. LCDA first aligns the distributions and avoids overlap between two domains. Then, LCDA exploits the local manifold structure to maintain the discriminative capability of the low-dimensional projection matrices. Finally, a robust constraint is utilized to preserve the robustness of the projection matrices. The proposed LCDA not only avoids overlap between different classes but also explores the local manifold information. Experiment results on the medical membranous nephropathy hyperspectral dataset demonstrate that the proposed LCDA has better performance than other relevant feature extraction methods.