Low-dimensional Subspace Research Articles

An Integer Programming Approach to Subspace Clustering with Missing Data

In the subspace clustering with missing data (SCMD) problem, we are given a collection of n partially observed d-dimensional vectors. The data points are assumed to be concentrated near a union of low-dimensional subspaces. The goal of SCMD is to cluster the vectors according to their subspace membership and recover the underlying basis, which can then be used to infer their missing entries. State-of-the-art algorithms for SCMD can fail on instances with a high proportion of missing data, with full-rank data, or if the underlying subspaces are similar to each other. We propose a novel integer programming approach for SCMD. The approach is based on dynamically determining a set of candidate subspaces and optimally assigning points to selected subspaces. The problem structure is identical to the classical facility-location problem, with subspaces playing the role of facilities and data points playing that of customers. We propose a column-generation approach for identifying candidate subspaces combined with a Benders decomposition approach for solving the linear programming relaxation of the formulation. An empirical study demonstrates that the proposed approach can achieve better clustering accuracy than state-of-the-art methods when the data are high rank, the percentage of missing data is high, or the subspaces are similar. Funding: Support for this research was provided by American Family Insurance through a research partnership with the University of Wisconsin–Madison’s Data Science Institute. Supplemental Material: The online appendix is available at https://doi.org/10.1287/ijoo.2023.0020 .

Environmental sound classification using raw-audio based ensemble framework

Environmental sound classification (ESC) aims to automatically recognize audio recordings from the underlying environment, such as "urban park" or "city centre". Most of the existing methods for ESC use hand-crafted time-frequency features such as log-mel spectrogram to represent audio recordings. However, the hand-crafted features rely on transformations that are defined beforehand and do not consider the variability in the environment due to differences in recording conditions or recording devices. To overcome this, we present an alternative representation framework by leveraging a pre-trained convolutional neural network, SoundNet, trained on a large-scale audio dataset to represent raw audio recordings. We observe that the representations obtained from the intermediate layers of SoundNet lie in low-dimensional subspace. However, the dimensionality of the low-dimensional subspace is not known. To address this, an automatic compact dictionary learning framework is utilized that gives the dimensionality of the underlying subspace. The low-dimensional embeddings are then aggregated in a late-fusion manner in the ensemble framework to incorporate hierarchical information learned at various intermediate layers of SoundNet. We perform experimental evaluation on publicly available DCASE 2017 and 2018 ASC datasets. The proposed ensemble framework improves performance between 1 and 4 percentage points compared to that of existing time-frequency representations.

Light field images super-resolution method based on hybrid low-dimensional spatial–angular interaction feature and linear complementation epipolar feature

Light-field (LF) cameras record 4D information about the intensity and angle of light, but limited by the resolution of the sensor, LF images require a trade-off between spatial and angular resolution, which results in generally low spatial resolution for LF images. We work on spatial super-resolution reconstruction (SR) of LF images, but due to the high dimensionality of 4D LF, it is difficult to process it directly, so we decompose 4D LF into low-dimensional sub-spaces. Since the 4D features of interest to different low-dimensional sub-spaces are different, we design a hybrid network structure—HSAESR, which processes multiple low-dimensional sub-spaces individually to adequately model the texture information in the 4D LF. HSAESR is designed to reconstruct Macro Pixel Images (MacPI) and Extremely Planar Images (EPI) respectively, and the final reconstruction results are weighted and fused. For MacPI, a CNN-based feature interaction module is designed using a novel asymptotic ’grid’ interaction approach to fully utilize the correlation between the 2D spatial dimension and the 2D angular dimension sub-spaces in 4D LF. For EPI, a Transformer-based epipolar feature linear complementation module is designed learning global and non-global multi-directional epipolar feature correlations, which learns feature linear correlation factor (α) and linear bias weight (β) through the correlation of non-global epipolar features, and corrects and complements the global epipolar correlation features through feature linear complementation. Extensive experiments are conducted on five datasets, the results show that HSAESR further improves the performance of LF images spatial SR, the reconstruction results are better than the comparison methods.

On-the-fly spectral unmixing based on Kalman filtering

This work introduces an on-the-fly (i.e., online) linear spectral unmixing method which is able to sequentially analyze spectral data acquired on a spectrum-by-spectrum basis. After deriving a sequential counterpart of the conventional linear mixing model, the proposed approach recasts the linear unmixing problem into a linear state-space estimation framework. Under Gaussian noise and state models, the estimation of the pure spectra can be efficiently conducted by resorting to Kalman filtering. Interestingly, it is shown that this Kalman filter can operate in a lower-dimensional subspace to lighten the computational burden of the overall unmixing procedure. Experimental results obtained on synthetic and real Raman data sets show that this Kalman filter-based method offers a convenient trade-off between unmixing accuracy and computational efficiency, which is crucial for operating in an on-the-fly setting. The proposed method constitutes a valuable building block for benefiting from acquisition and processing frameworks recently proposed in the microscopy literature, which are motivated by practical issues such as reducing acquisition time and avoiding potential damages being inflicted to photosensitive samples. The code associated with the numerical illustrations reported in this paper is freely available online at https://github.com/HKouakou/KF-OSU.

Inferring Covariance Structure from Multiple Data Sources via Subspace Factor Analysis

Factor analysis provides a canonical framework for imposing lower-dimensional structure such as sparse covariance in high-dimensional data. High-dimensional data on the same set of variables are often collected under different conditions, for instance in reproducing studies across research groups. In such cases, it is natural to seek to learn the shared versus condition-specific structure. Existing hierarchical extensions of factor analysis have been proposed, but face practical issues including identifiability problems. To address these shortcomings, we propose a class of SUbspace Factor Analysis (SUFA) models, which characterize variation across groups at the level of a lower-dimensional subspace. We prove that the proposed class of SUFA models lead to identifiability of the shared versus group-specific components of the covariance, and study their posterior contraction properties. Taking a Bayesian approach, these contributions are developed alongside efficient posterior computation algorithms. Our sampler fully integrates out latent variables, is easily parallelizable and has complexity that does not depend on sample size. We illustrate the methods through application to integration of multiple gene expression datasets relevant to immunology.

Deriving Three-Outcome Permutationally Invariant Bell Inequalities.

We present strategies to derive Bell inequalities valid for systems composed of many three-level parties. This scenario is formalized by a Bell experiment with N observers, each of which performs one out of two possible three-outcome measurements on their share of the system. As the complexity of the set of classical correlations prohibits its full characterization in this multipartite scenario, we consider its projection to a lower-dimensional subspace spanned by permutationally invariant one- and two-body observables. This simplification allows us to formulate two complementary methods for detecting nonlocality in multipartite three-level systems, both having a complexity independent of N. Our work can have interesting applications in the detection of Bell correlations in paradigmatic spin-1 models, as well as in experiments with solid-state systems or atomic ensembles.

The fast committor machine: Interpretable prediction with kernels.

In the study of stochastic systems, the committor function describes the probability that a system starting from an initial configuration x will reach a set B before a set A. This paper introduces an efficient and interpretable algorithm for approximating the committor, called the "fast committor machine" (FCM). The FCM uses simulated trajectory data to build a kernel-based model of the committor. The kernel function is constructed to emphasize low-dimensional subspaces that optimally describe the A to B transitions. The coefficients in the kernel model are determined using randomized linear algebra, leading to a runtime that scales linearly with the number of data points. In numerical experiments involving a triple-well potential and alanine dipeptide, the FCM yields higher accuracy and trains more quickly than a neural network with the same number of parameters. The FCM is also more interpretable than the neural net.

Greedy identification of latent dynamics from parametric flow data

Projection-based reduced-order models (ROMs) play a crucial role in simplifying the complex dynamics of fluid systems. Such models are achieved by projecting the Navier-Stokes equations onto a lower-dimensional subspace while preserving essential dynamics. However, this approach requires prior knowledge of the underlying high-fidelity model, limiting its effectiveness when applied to black-box data. This article introduces a novel, non-intrusive, data-driven method–Greedy Identification of Latent Dynamics (GILD)–for constructing parametric fluid ROMs. Unlike traditional methods, GILD constructs models directly from data, without relying on specific high-fidelity model information. It also employs interpolation within the manifold R∗N×q/Oq to accommodate parameter variability. Numerical experiments on various fluid dynamics scenarios, including lid-driven cavity flow, flow past a cylinder with varying Reynolds number, and Ahmed body flow with variable geometry, demonstrate GILD’s robust performance across both training and unseen parameter values. GILD’s ability to accurately capture system dynamics and its adaptability to diverse data sources highlight its potential as a powerful tool for constructing parametric reduced-order models in an easy and general way for complex fluid dynamics and beyond.

Stereologically adapted Crofton formulae for tensor valuations

The classical Crofton formula explains how intrinsic volumes of a convex body K in n-dimensional Euclidean space can be obtained from integrating a measurement function at sections of K with invariantly moved affine flats. We generalize this idea by constructing stereologically adapted Crofton formulae for translation invariant Minkowski tensors, expressing a prescribed tensor valuation as an invariant integral of a measurement function of section profiles with flats. The measurement functions are weighed sums of powers of the metric tensor times Minkowski valuations. The weights are determined explicitly from known Crofton formulae using Zeilberger's algorithm. The main result is an exhaustive set of measurement functions where the invariant integration is over flats.With the main result at hand, a Blaschke-Petkantschin formula allows us to establish new measurement functions valid when the invariant integration over flats is replaced by an invariant integration over subspaces containing a fixed subspace of lower dimension. Likewise, a stereologically adapted Crofton formula valid in the scheme of vertical sections is constructed. Only some special cases of this result have been stated explicitly before, with even the three-dimensional case yielding a new stereological formula. Here, we obtain new vertical section formulae for the surface tensors of even rank.

Interacting crystalline topological insulators in two-dimensions with time-reversal symmetry

Topology is routinely used to understand the physics of electronic insulators. However, for strongly interacting electronic matter, such as Mott insulators, a comprehensive topological characterization is still lacking. When their ground state only contains short-range entanglement and does not break symmetries spontaneously, they generically realize crystalline fermionic symmetry-protected topological phases (cFSPTs), supporting gapless modes at the boundaries or at the lattice defects. Here, we provide an exhaustive classification of cFSPTs in two dimensions with U(1) charge-conservation and spinful time-reversal symmetries, namely, those generically present in spin-orbit coupled insulators, for any of the 17 wallpaper groups. It has been shown that the classification of cFSPTs can be understood from appropriate real-space decorations of lower-dimensional subspaces, and we expose how these relate to the Wyckoff positions of the lattice. We find that all nontrivial one-dimensional decorations require electronic interactions. Furthermore, we provide model Hamiltonians for various decorations, and discuss the signatures of cFSPTs. This classification paves the way to further explore topological interacting insulators, providing the backbone information in generic model systems and ultimately in experiments. Published by the American Physical Society 2024

Understanding and mitigating dimensional collapse of Graph Contrastive Learning: A non-maximum removal approach

Graph Contrastive Learning (GCL) generates graph-level embeddings by maximizing Mutual Information between different augmented views of the same graph (positive pairs), and shows promising performance in graph representation learning (GRL) without the supervision of manual annotations. However, GCL suffers from a dimensional collapse problem, i.e., embedding vectors reside in a restricted low-dimensional subspace, curtailing the expressiveness of certain embedding dimensions. In this paper, we present a theoretical analysis identifying the smoothing effect of graph pooling and graph convolution’s implicit regularization as principal causes of dimension collapse in graph contrastive learning. To mitigate the above effects, we propose a Non-Maximum Removal Graph Contrastive Learning (nmrGCL) approach, which removes “prominent” dimensions (those significantly contributing to the similarity measure) for positive pairs within the pretext task. Comprehensive experiments on multiple benchmark datasets are conducted and the results show that the proposed nmrGCL outperforms the state-of-the-art methods.

Flexible control of sequence working memory in the macaque frontal cortex

To memorize a sequence, one must serially bind each item to its rank order. How the brain controls a given input to bind its associated order in sequence working memory (SWM) remains unexplored. Here, we investigated the neural representations underlying SWM control using electrophysiological recordings in the frontal cortex of macaque monkeys performing forward and backward SWM tasks. Separate and generalizable low-dimensional subspaces for sensory and memory information were found within the same frontal circuitry, and SWM control was reflected in these neural subspaces' organized dynamics. Each item at each rank was sequentially entered into a common sensory subspace and, depending on forward or backward task requirement, flexibly and timely sent into rank-selective SWM subspaces. Neural activity in these SWM subspaces faithfully predicted the recalled item and order information in single error trials. Thus, compositional neural population codes with well-orchestrated dynamics in frontal cortex support the flexible control of SWM.

Data-driven transient lift attenuation for extreme vortex gust–airfoil interactions

We present a data-driven feedforward control to attenuate large transient lift experienced by an airfoil disturbed by an extreme level of discrete vortex gust. The current analysis uses a nonlinear machine-learning technique to compress the high-dimensional flow dynamics onto a low-dimensional manifold. While the interaction dynamics between the airfoil and extreme vortex gust are parametrized by its size, gust ratio and position, the wake responses are well captured on this simple manifold. The effect of extreme vortex disturbance about the undisturbed baseline flows can be extracted in a physically interpretable manner. Furthermore, we call on phase-amplitude reduction to model and control the complex nonlinear extreme aerodynamic flows. The present phase-amplitude reduction model reveals the sensitivity of the dynamical system in terms of the phase shift and amplitude change induced by external forcing with respect to the baseline periodic orbit. By performing the phase-amplitude analysis for a latent dynamical model identified by sparse regression, the sensitivity functions of low-dimensionalized aerodynamic flows for both phase and amplitude are derived. With the phase and amplitude sensitivity functions, optimal forcing can be determined to quickly suppress the effect of extreme vortex gusts towards the undisturbed states in a low-order space. The present optimal flow modification built upon the machine-learned low-dimensional subspace quickly alleviates the impact of transient vortex gusts for a variety of extreme aerodynamic scenarios, providing a potential foundation for flight of small-scale air vehicles in adverse atmospheric conditions.

Contrastive subspace distribution learning for novel category discovery in high-dimensional visual data

Discovering novel visual categories, particularly in recognition tasks, is a prominent area of research in artificial intelligence. Visual data, such as images and videos, is inherently high-dimensional and they often exhibit low-dimensional subspace structures, making the task of uncovering novel low-dimensional subspace structures a substantial challenge. To address this challenge, we develop a novel subspace discovery method, consisting of four steps. First, we introduce a base-expressive network (BENet) designed to learn coefficients that express the relationships between samples and bases. Additionally, these coefficients are normalized as a probability distribution using the softmax function. Second, to enhance this process, we propose a Contrastive Subspace Distribution Learning (CSDL) technique, utilizing Rényi Divergence. CSDL combines self-supervised and supervised contrastive distances between the normalized coefficients, considering all samples and labeled samples, respectively. Third, the base-expressive coefficients are subsequently computed by the learned SENet, which effectively transfers subspace information from labeled samples to unlabeled ones. Finally, we employ spectral clustering on the predicted coefficients to automatically assign categories to unlabeled samples. Extensive experimental results demonstrate that our approach surpasses related methods in the realms of category discovery, unsupervised clustering, and semi-supervised (subspace) clustering.

A type of multigrid method for semilinear elliptic problems based on symmetric interior penalty discontinuous Galerkin method

AbstractThis article introduces a new kind of multigrid approach for semilinear elliptic problems, which is based on the symmetric interior penalty discontinuous Galerkin (SIPDG) method. We first give an optimal error estimate of the SIPDG method for the problem. Then, we design a type of multigrid method, which is called the multilevel correction method, and derive a priori error estimates. The primary idea of this method is to take the solution of the semilinear problem and utilize it to establish a sequence of solutions for associated linear boundary value problem on discontinuous finite element spaces and a newly defined low dimensional augmented subspace. Lastly, numerical experiments are offered to confirm the suggested method's precision and effectiveness.

Projective Incomplete Multi-View Clustering.

Due to the rapid development of multimedia technology and sensor technology, multi-view clustering (MVC) has become a research hotspot in machine learning, data mining, and other fields and has been developed significantly in the past decades. Compared with single-view clustering, MVC improves clustering performance by exploiting complementary and consistent information among different views. Such methods are all based on the assumption of complete views, which means that all the views of all the samples exist. It limits the application of MVC, because there are always missing views in practical situations. In recent years, many methods have been proposed to solve the incomplete MVC (IMVC) problem and a kind of popular method is based on matrix factorization (MF). However, such methods generally cannot deal with new samples and do not take into account the imbalance of information between different views. To address these two issues, we propose a new IMVC method, in which a novel and simple graph regularized projective consensus representation learning model is formulated for incomplete multi-view data clustering task. Compared with the existing methods, our method not only can obtain a set of projections to handle new samples but also can explore information of multiple views in a balanced way by learning the consensus representation in a unified low-dimensional subspace. In addition, a graph constraint is imposed on the consensus representation to mine the structural information inside the data. Experimental results on four datasets show that our method successfully accomplishes the IMVC task and obtain the best clustering performance most of the time. Our implementation is available at https://github.com/Dshijie/PIMVC.

Structural generation by inverse transformation using principal component analysis enhances conformational sampling of protein

Abstract Molecular dynamics (MD) simulations are frequently used to elucidate the molecular mechanisms underlying protein behavior. Based on a conformational search with MD simulations, protein structures rich in high-dimensional data can be quantitatively evaluated in free-energy landscapes (FELs). Generally, FELs are defined in low-dimensional subspaces spanned by reaction coordinates to characterize biological functions. When calculating FELs of proteins, principal component analysis (PCA) is particularly useful for capturing large-amplitude motions via dimensionality reduction into low-dimensional subspaces. In this study, to efficiently calculate FELs, a simple and convenient method is proposed by accelerating conformational search in a PCA subspace, which is achieved by quick generation of protein configurations. Specifically, inverse transformation driven by PCA facilitates the quick generation of diverse protein configurations from arbitrary grids in a defined PCA subspace. In our conformational search, a set of newly generated configurations serves as initial structures for multiple MD simulations, enabling one to calculate FELs of proteins by building Markov state models from their multiple trajectories. In conclusion, the conformational search from protein configurations broadly distributed in a PCA subspace accelerates FEL calculations, which supports a comprehensive approach to understanding collective protein dynamics.

High-resolution enhanced cross-subspace fusion network for light field image superresolution

Light field (LF) images offer abundant spatial and angular information, therefore, the combination of which is beneficial in the performance of LF image superresolution (LF image SR). Currently, existing methods often decompose the 4D LF data into low-dimensional subspaces for individual feature extraction and fusion for LF image SR. However, the performance of these methods is restricted because of lacking effective correlations between subspaces and missing out on crucial complementary information for capturing rich texture details. To address this, we propose a cross-subspace fusion network for LF spatial SR (i.e., CSFNet). Specifically, we design the progressive cross-subspace fusion module (PCSFM), which can progressively establish cross-subspace correlations based on a cross-attention mechanism to comprehensively enrich LF information. Additionally, we propose a high-resolution adaptive enhancement group (HR-AEG), which preserves the texture and edge details in the high resolution feature domain by employing a multibranch enhancement method and an adaptive weight strategy. The experimental results demonstrate that our approach achieves highly competitive performance on multiple LF datasets compared to state-of-the-art (SOTA) methods.

Domain Decomposition and Model Order Reduction for Electromagnetic Field Simulations in Carbon Fiber Composite Materials

The computation of the electric field in composite materials at the microscopic scale results in an immense number of degrees of freedom. Consequently, this often leads to prohibitively long computation times and extensive memory requirements, making direct computation impractical. In this study, one employs an innovative approach that integrates domain decomposition and model order reduction to retain local information while significantly reducing computation time. Domain decomposition allows for the division of the computational domain into smaller, more manageable subdomains, enabling parallel processing and reducing the overall complexity of the problem. Model order reduction further enhances this by approximating the solution in a lower-dimensional subspace, thereby minimising the number of unknown variables that need to be computed. Comparative analysis between the results obtained from the reduced model and those from direct resolution demonstrates that our method not only reduces computation time but also maintains accuracy. The new method effectively captures the essential characteristics of the electric field distribution in composite materials, ensuring that the local phenomena are accurately represented. This study provides a contribution to the field of computational electromagnetics by presenting a feasible solution to the challenges posed by the high computational demands of simulating composite materials at the microscopic scale. The proposed methodology offers a promising direction for future research and practical applications, enabling more efficient and accurate simulations of complex material systems.

Projected data assimilation using sliding window proper orthogonal decomposition

Prediction of the state evolution of complex high-dimensional nonlinear systems is challenging due to the nonlinear sensitivity of the evolution to small inaccuracies in the model. Data Assimilation (DA) techniques improve state estimates by combining model simulations with real-time data. Few DA techniques can simultaneously handle nonlinear evolution, non-Gaussian uncertainty, and the high dimension of the state. We recently proposed addressing these challenges using a POD technique that projects the physical and data models into a reduced-dimensional subspace. Proper Orthogonal Decomposition (POD) is a tool to extract spatiotemporal patterns (modes) that dominate the observed data. We combined the POD-based projection operator, computed in an offline fashion, with a DA scheme that models non-Gaussian uncertainty in lower dimensional subspace. If the model parameters change significantly during time evolution, the offline computation of the projection operators ceases to be useful. We address this challenge using a Sliding Window POD (SW-POD), which recomputes the projection operator based on a sliding subset of snapshots from the entire evolution. The physical model projection is updated dynamically in terms of modes and number of modes, and the data model projection is also chosen to promote a sparse approximation. We test the efficacy of this technique on a modified Lorenz'96 model (L96) with a time-varying forcing and compare it with the time-invariant offline projected algorithm. In particular, dynamically determined physical and data model projections decrease the Root Mean Squared Error (RMSE) and the resampling rate.