Low-dimensional Subspace Research Articles

Geometry of sequence working memory in macaque prefrontal cortex.

How the brain stores a sequence in memory remains largely unknown. We investigated the neural code underlying sequence working memory using two-photon calcium imaging to record thousands of neurons in the prefrontal cortex of macaque monkeys memorizing and then reproducing a sequence of locations after a delay. We discovered a regular geometrical organization: The high-dimensional neural state space during the delay could be decomposed into a sum of low-dimensional subspaces, each storing the spatial location at a given ordinal rank, which could be generalized to novel sequences and explain monkey behavior. The rank subspaces were distributed across large overlapping neural groups, and the integration of ordinal and spatial information occurred at the collective level rather than within single neurons. Thus, a simple representational geometry underlies sequence working memory.

Fusion of effective dimension reduction and discriminative dictionary learning for high-dimensional classification

High-dimensional classification is a hot topic of pattern recognition. However, its performance has much room for improvement due to environment uncertainties and other factors. For example, redundant information from high dimensional data, illumination, and occlusion may lead to the inferior classification performance. Additionally, some potentially useful features of high-dimensional data are not fully captured. To address these problems, we propose a fusion of effective dimension reduction and discriminative dictionary learning for high-dimensional classification. A novel dimension reduction approach is employed to map training samples into a low-dimensional space by using the distance and label information. Then in the low-dimensional subspace, we integrate atomic Fisher constraint and local constraint of coding coefficients into the dictionary learning. Thus the discriminative atoms are effectively extracted. Finally, experimental results prove the efficacy of the proposed model. Besides, the comparative experiments with AlexNet and VGG19 prove the superiority of our method over some deep learning approaches.

Reduced order modeling of nonlinear microstructures through Proper Orthogonal Decomposition

We apply the Proper Orthogonal Decomposition (POD) method for the efficient simulation of several scenarios undergone by Micro-Electro-Mechanical-Systems, involving nonlinearites of geometric and electrostatic nature. The former type of nonlinearity, associated to the large displacements of the devices, leads to polynomial terms up to cubic order that are reduced through exact projection onto a low-dimensional subspace spanned by the Proper Orthogonal Modes (POMs). On the contrary, electrostatic nonlinearities are modeled resorting to precomputed manifolds in terms of the amplitudes of the electrically active POMs. We extensively test the reliability of the assumed linear trial space in challenging applications focusing on resonators, micromirrors and arches also displaying internal resonances. We discuss several options to generate the matrix of snapshots using both classical time marching schemes and more advanced Harmonic Balance (HB) approaches. Furthermore, we propose a comparison between the periodic orbits computed with POD and the invariant manifold approximated with Direct Normal Form approaches, further stressing the reliability of the technique and its remarkable predictive capabilities, e.g., in terms of estimation of the frequency response function of selected output quantities of interest.

Balanced Hodge Laplacians optimize consensus dynamics over simplicial complexes.

Despite the vast literature on network dynamics, we still lack basic insights into dynamics on higher-order structures (e.g., edges, triangles, and more generally, k-dimensional "simplices") and how they are influenced through higher-order interactions. A prime example lies in neuroscience where groups of neurons (not individual ones) may provide building blocks for neurocomputation. Here, we study consensus dynamics on edges in simplicial complexes using a type of Laplacian matrix called a Hodge Laplacian, which we generalize to allow higher- and lower-order interactions to have different strengths. Using techniques from algebraic topology, we study how collective dynamics converge to a low-dimensional subspace that corresponds to the homology space of the simplicial complex. We use the Hodge decomposition to show that higher- and lower-order interactions can be optimally balanced to maximally accelerate convergence and that this optimum coincides with a balancing of dynamics on the curl and gradient subspaces. We additionally explore the effects of network topology, finding that consensus over edges is accelerated when two-simplices are well dispersed, as opposed to clustered together.

Discriminant Analysis under f-Divergence Measures.

In statistical inference, the information-theoretic performance limits can often be expressed in terms of a statistical divergence between the underlying statistical models (e.g., in binary hypothesis testing, the error probability is related to the total variation distance between the statistical models). As the data dimension grows, computing the statistics involved in decision-making and the attendant performance limits (divergence measures) face complexity and stability challenges. Dimensionality reduction addresses these challenges at the expense of compromising the performance (the divergence reduces by the data-processing inequality). This paper considers linear dimensionality reduction such that the divergence between the models is maximally preserved. Specifically, this paper focuses on Gaussian models where we investigate discriminant analysis under five f-divergence measures (Kullback–Leibler, symmetrized Kullback–Leibler, Hellinger, total variation, and ). We characterize the optimal design of the linear transformation of the data onto a lower-dimensional subspace for zero-mean Gaussian models and employ numerical algorithms to find the design for general Gaussian models with non-zero means. There are two key observations for zero-mean Gaussian models. First, projections are not necessarily along the largest modes of the covariance matrix of the data, and, in some situations, they can even be along the smallest modes. Secondly, under specific regimes, the optimal design of subspace projection is identical under all the f-divergence measures considered, rendering a degree of universality to the design, independent of the inference problem of interest.

Meta-Pixel-Driven Embeddable Discriminative Target and Background Dictionary Pair Learning for Hyperspectral Target Detection

In hyperspectral target detection, the spectral high-dimensionality, variability, and heterogeneity will pose great challenges to the accurate characterizations of the target and background. To alleviate the problems, we propose a Meta-pixel-driven Embeddable Discriminative target and background Dictionary Pair (MEDDP) learning model by combining low-dimensional embeddable subspace projection and the discriminative target and background dictionary pair learning. In MEDDP, the meta-pixel set is built by taking the merits of homogeneous superpixel segmentation and the local manifold affinity structures, which can significantly reduce the influence of spectral variability and find the most typical and informative prototype spectral signature. Afterward, an embeddable discriminative dictionary pair learning model is established to learn a target and background dictionary pair based on the structural incoherent constraint with embeddable subspace projection. The proposed joint learning strategy can reduce the high-dimensional redundant information and simultaneously enhance the discrimination and compactness of the target and background dictionaries. The proposed MEDDP model is solved by an iterative and alternate optimization algorithm and applied with the meta-pixel-level target detection method. Experimental results on four benchmark HSI datasets indicate that the proposed method can consistently yield promising performance in comparison with some state-of-the-art target detectors.

IntroVAC: Introspective Variational Classifiers for learning interpretable latent subspaces

Learning useful representations of complex data has been the subject of extensive research for many years. In particular, with the diffusion of complex Deep Learning-based approaches in engineering applications, the possibility to interpret, to a certain degree, model predictions is of fundamental importance for both the model users and developers. In the context of Deep Neural Networks, Variational Autoencoders have gained lots of attention since they provide an explicit model of the data distribution based on an encoder/decoder architecture which is able to both generate images and encode them in a low-dimensional subspace. However, the latent space is not easily interpretable and the generation capabilities show some limitations since images typically look blurry and lack details. In this paper, we propose the Introspective Variational Classifier (IntroVAC), a model that learns interpretable latent subspaces by exploiting information from an additional label and provides improved image quality thanks to an adversarial training strategy. We show that IntroVAC is able to learn meaningful directions in the latent space enabling fine-grained manipulation of image attributes. We validated our approach on the CelebA dataset. When compared with standard Variational Autoencoder Classifiers, the proposed approach outperform them by achieving a Frechét Inception Distance of 25.5 versus a value of 63.9.

Joint projection learning and structured analysis-synthesis dictionary pair learning for pattern classification

Analysis-synthesis dictionary pair learning has attracted much attention in the field of pattern classification. To reduce the negative effect of trivial information contained in raw training samples and improve the computation efficiency, most existing dictionary pair learning methods first learn a projection matrix to project raw training samples into a low-dimensional subspace and then use the dimensionality-reduced samples for dictionary pair learning. However, the separation of projection learning (PL) and dictionary pair learning may make the dimensionality-reduced training samples not fit well for dictionary pair learning. To address this issue, we proposed a joint PL and structured analysis-synthesis dictionary pair learning (PLSDPL) method for pattern classification. Specifically, PLSDPL integrates PL and dictionary pair learning into a unified framework, so the dimensionality-reduced training samples are more suitable for the structured analysis-synthesis dictionary pair learning, thus making the learned analysis-synthesis dictionary pair have powerful discrimination capability. In addition, PLSDPL also imposes a low-rank constraint on each synthesis sub-dictionary to reduce the negative effect of noise contained in training samples. Experimental results on several image datasets show that PLSDPL is effective for pattern classification tasks.

Autoencoder-aided analysis of low-dimensional Hilbert spaces

Keywords: feedforward autoencoder, low-dimensional Hilbert spaces, numerical ground-state estimation We study the applicability of feedforward autoencoders in determining the ground state of a quantum system from a noisy signal provided in a form of random superpositions sampled from a low-dimensional subspace of the system’s Hilbert space. The proposed scheme relies on a minimum set of assumptions: the presence of a finite number of orthogonal states in the samples and a weak statistical dominance of the targeted ground state. The provided data is compressed into a two-dimensional feature space and subsequently analyzed to determine the optimal approximation to the true ground state. The scheme is applicable to single- and many-particle quantum systems as well as in the presence of magnetic frustration.

Low Dimensional Trajectory Hypothesis is True: DNNs Can Be Trained in Tiny Subspaces.

Deep neural networks (DNNs) usually contain massive parameters, but there is redundancy such that it is guessed that they could be trained in low-dimensional subspaces. In this paper, we propose a Dynamic Linear Dimensionality Reduction (DLDR) based on the low-dimensional properties of the training trajectory. The reduction method is efficient, supported by comprehensive experiments: optimizing DNNs in 40-dimensional spaces can achieve comparable performance as regular training over thousands or even millions of parameters. Since there are only a few variables to optimize, we develop an efficient quasi-Newton-based algorithm, obtain robustness to label noise, and improve the performance of well-trained models, which are three follow-up experiments that can show the advantages of finding such low-dimensional subspaces. The code is released (Pytorch: https://github.com/nblt/DLDR and Mindspore: https://gitee.com/mindspore/docs/tree/r1.6/docs/sample_code/dimension_reduce_training).

High-dimensional sufficient dimension reduction through principal projections

We develop in this work a new dimension reduction method for high-dimensional settings. The proposed procedure is based on a principal support vector machine framework where principal projections are used in order to overcome the non-invertibility of the covariance matrix. Using a series of equivalences we show that one can accurately recover the central subspace using a projection on a lower dimensional subspace and then applying an ℓ1 penalization strategy to obtain sparse estimators of the sufficient directions. Based next on a desparsified estimator, we provide an inferential procedure for high-dimensional models that allows testing for the importance of variables in determining the sufficient direction. Theoretical properties of the methodology are illustrated and computational advantages are demonstrated with simulated and real data experiments.

Closed-Loop Subspace Identification for Stable/ Unstable Systems Using Data Compression and Nuclear Norm Minimization

This paper provides a subspace method for closed-loop identification, which clearly specifies the model order from noisy measurement data. The method can handle long I/O data of the target system to be noise-tolerant and determine the model order via nuclear norm minimization. First, the proposed method compresses the long data by projecting them to an appropriate low dimensional subspace, then obtains a low order model whose order is specified by a combination of data compression and nuclear norm minimization. Its effectiveness is demonstrated through detailed numerical examples.

A FULLY BAYESIAN GRADIENT-FREE SUPERVISED DIMENSION REDUCTION METHOD USING GAUSSIAN PROCESSES

Modern day engineering problems are ubiquitously characterized by sophisticated computer codes that map parameters or inputs to an underlying physical process. In other situations, experimental setups are used to model the physical process in a laboratory, ensuring high precision while being costly in materials and logistics. In both scenarios, only limited amount of data can be generated by querying the expensive information source at a finite number of inputs or designs. This problem is compounded further in the presence of a high-dimensional input space. State-of-the-art parameter space dimension reduction methods, such as active subspace, aim to identify a subspace of the original input space that is sufficient to explain the output response. These methods are restricted by their reliance on gradient evaluations or copious data, making them inadequate to expensive problems without direct access to gradients. The proposed methodology is gradient-free and fully Bayesian, as it quantifies uncertainty in both the low-dimensional subspace and the surrogate model parameters. This enables a full quantification of epistemic uncertainty and robustness to limited data availability. It is validated on multiple datasets from engineering and science and compared to two other state-of-the-art methods based on four aspects: a) recovery of the active subspace, b) deterministic prediction accuracy, c) probabilistic prediction accuracy, and d) training time. The comparison shows that the proposed method improves the active subspace recovery and predictive accuracy, in both the deterministic and probabilistic sense, when only few model observations are available for training, at the cost of increased training time.

Spatial-Aware Collaboration–Competition Preserving Graph Embedding for Hyperspectral Image Classification

Recently, graph-based discriminant analysis has drawn much attention in representing a high-dimensional hyperspectral data set using a low-dimensional subspace by defining high-dimensional data structure on a graph. Obtaining optimal representation coefficients for classification purposes are the key in such methods. A closed form solution can be found to solve the problem related to collaborative representation using labeled samples, which offers computational efficiency. There exists an unsupervised approach of collaboration preserving graph embedding (CPGE) for dimensionality reduction (DR), and its performance is further enhanced by imposing locality-preserving constraint in the method called collaboration–competition preserving graph embedding (CCPGE). In this letter, we introduce spatial-aware collaboration–competitive preserving graph embedding with Tikhonov (SaCCPGT) by imposing a spatial regularization term in the objective function of CCPGE with Tikhonov regularization. In this way, spectral and spatial information can be utilized in a closed form solution in the proposed method. Experimental results on different hyperspectral data sets demonstrate the superior performance of the proposed SaCCPGT in comparison to state-of-the-art graph-based discriminant analysis approaches for DR.

Blind Two-Dimensional Super-Resolution and Its Performance Guarantee

In this work, we study the problem of identifying the parameters of a linear system from its response to multiple unknown input waveforms. We assume that the system response, which is the only given information, is a scaled superposition of time-delayed and frequency-shifted versions of the unknown waveforms. Such kind of problem is severely ill-posed and does not yield a unique solution without introducing further constraints. To fully characterize the linear system, we assume that the unknown waveforms lie in a common known low-dimensional subspace that satisfies certain randomness and concentration properties. Then, we develop a blind two-dimensional (2D) super-resolution framework that applies to a large number of applications such as radar imaging, image restoration, and indoor source localization. In this framework, we show that under a minimum separation condition between the time-frequency shifts, all the unknowns that characterize the linear system can be recovered precisely and with very high probability provided that a lower bound on the total number of the observed samples is satisfied. The proposed framework is based on 2D atomic norm minimization problem which is shown to be reformulated and solved efficiently via semidefinite programming. Simulation results that confirm the theoretical findings of the paper are provided.

A Structural Subspace Clustering Approach for Hyperspectral Band Selection

Band selection, which removes irrelevant bands from hyperspectral images (HSIs) and keeps essential spectral information contained in a relatively few bands, allows huge savings in data storage, computation time, and imaging hardware. In this article, we propose a novel structural subspace clustering (STSC) method for hyperspectral band selection, which leverages the self-representation property of data and structural prior information to learn the cluster structure of bands. Particularly, we propose a general clustering model where the coarse coefficients matrix derived from a self-representation model is decomposed as a combination of a desirable coefficients matrix and a sparse matrix. This strategy adaptively adjusts the coarse coefficients matrix to learn the intrinsic data structure in low-dimensional subspaces. To guide this learning process, we introduce a structural regularization approach which makes use of the prior information about local and global properties of spectral bands. Moreover, we incorporate also prior knowledge about the dictionary, which demonstrates to yield a better clustering performance. We develop an adaptive method to estimate the number of selected bands by analyzing eigenvalue gaps of Laplacian matrix. To solve the resulting model, an efficient algorithm based on alternating direction method of multipliers (ADMMs) is developed. Extensive experiments on benchmark HSIs show that our method outperforms the state-of-the-art band selection methods.

Unmixing-Based PAN-Guided Fusion Network for Hyperspectral Imagery

The hyperspectral image (HSI) has been widely used in many applications due to its fruitful spectral information. However, the limitation of imaging sensors has reduced its spatial resolution that causes detail loss. One solution is to fuse the low spatial resolution HSI (LR-HSI) and the panchromatic (PAN) image with inverse features to get the high-resolution HSI (HR-HSI). Most of the existing fusion methods just focus on small fusion ratios like 4 or 6, which might be impractical for some large ratios’ HSI and PAN image pairs. Moreover, the ill-posedness of restoring detail information in HSI with hundreds of bands from PAN image with only one band has not been solved effectively, especially under large fusion ratios. Therefore, a lightweight unmixing-based PAN-guided fusion network (Pgnet) is proposed to mitigate this ill-posedness and improve the fusion performance significantly. Note that the fusion process of the proposed network is under the projected low-dimensional abundance subspace with an extremely large fusion ratio of 16. Furthermore, based on the linear and nonlinear relationships between the PAN intensity and abundance, an interpretable PAN detail inject network (PDIN) is designed to inject the PAN details into the abundance feature efficiently. Comprehensive experiments on simulated and real datasets demonstrate the superiority and generality of our method over several state-of-the-art (SOTA) methods qualitatively and quantitatively (the codes in pytorch and paddle versions and dataset could be available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/rs-lsl/Pgnet</uri> ).

Feature-based ensemble history matching in a fractured carbonate reservoir using time-lapse deep electromagnetic tomography

Carbonate reservoirs typically exhibit very complex geological structures and are characterized by flow dynamics primarily occurring in fractures. The intricate network of fractures as well as their interconnectedness may lead to unexpected flow patterns and uneven sweep efficiency. Determining reservoir properties of both matrix and fracture channels is quintessential for accurately tracking the fluid front movement in the reservoir, optimizing sweep efficiency, and maximizing hydrocarbon production. In this study, we showcase the application of a feature-oriented ensemble-based history matching workflow to a complex fractured carbonate reservoir box model, focusing on the use of formation resistivity tomography data that are usually inferred from deep crosswell electromagnetic (EM) surveys. Compared with the production data that are commonly used in history matching, deep EM measurements provide additional information about the spatial distribution of subsurface reservoir properties in the interwell volumes by exploiting the strong resistivity contrast between water and hydrocarbons. A hybrid parameterization approach is used to represent the multiscale fracture distribution in which the spatial distribution of small-scale fractures is modelled by a truncated Gaussian simulation method. A large number (over one million) of uncertain model parameters including reservoir matrix and fracture properties as well as Archie's parameters are identified and updated by an iterative ensemble smoother. For an efficient integration of the high-dimensional and noisy EM tomography data, the boundary or contour information extracted from the EM resistivity field is instead assimilated through a distance parameterization approach. A modified bootstrap-based localization is proposed to regularize the model updates adaptively during the iteration to reduce sampling errors. Especially, to improve the computational efficiency in dealing with the large dimensions of both data and model parameters, the localization is implemented in a projected low-dimensional data subspace. Experimental results demonstrate the applicability and efficiency of the developed workflow for reservoir history matching in more realistic model settings. The comparative case study also illustrates the significance of jointly incorporating multiple sources of data for better quantification of model uncertainty, and the great potential of deep EM data for enhancing the characterization of complex fractured carbonate reservoirs.

MAXVAR-Based Distributed Correlation Estimation in a Wireless Sensor Network

The performance of most array signal processing tasks relies on the presence of correlation between sensor signals. In a wireless sensor network, where sensor nodes are spread out over a relatively large area, it is useful to identify nodes observing similar sensor signals and hence common phenomenons, for example to partition the network according to the observed latent signals and corresponding correlation structure. This can be achieved via the so-called MAXVAR formulation of generalized canonical correlation analysis, which finds a low-dimensional subspace that highlights correlated signal components between multiple nodes' observed signal subspaces. The classical procedure for computing the solutions of MAXVAR consists in performing a generalized eigenvalue decomposition after collecting all the sensors' signals at a fusion center. However, this typically incurs high communication and computational costs. In this paper, we describe a low communication and computational cost distributed algorithm that computes the solutions of MAXVAR without aggregating the nodes' observations at a central location. We show how a subset of those solutions can be used locally by each node to estimate the global correlation structure across all nodes in the network, thereby allowing any node to evaluate the presence of correlated signals at any other node, even if no direct link is shared. We prove the convergence of the algorithm and validate our method for estimating the correlation structure via simulations.

Exploring Nonlocal Group Sparsity Under Transform Learning for Hyperspectral Image Denoising

Hyperspectral image (HSI) denoising has been regarded as an effective and economical preprocessing step in data subsequent applications. Recent nonlocal low-rank approximation on each full band patch group has demonstrated their superiority for HSI denoising. These methods, however, directly design the low-rank regularization to the grouped patch image itself (i.e., original domain), which ignores the spatial information of the grouped patch image and cannot explores the potential structure. To address these issues, this paper proposes a nonlocal group sparsifying transform learning method (dubbed TLNLGS) for HSI denoising. Motivated by the global spectral correlation in the HSI, we firstly impose a certain low-dimensional subspace hypothesis over the HSI to prevent the heavy computation burden with the spectral band increases, and then explore a discriminatively intrinsic nonlocal group sparse prior of the reduced image by transform model. The learned group sparse prior can not only excavate the nonlocal self-similarity as recent nonlocal low-rank approximation methods but also preserve the local spatial smooth structure of the image. Moreover, compared with the fixed transform domain (e.g., gradient and discrete cosine transformation domains), the transform learning scheme can improve the sparse representation ability. An efficient block coordinate descent (BCD) algorithm is developed to solve the proposed model. Extensive experiments, including simulated and real HSI datasets, indicate the superiority of the proposed TLNLGS method over the state-of-the-art HSI denoising approaches.