Low-rank Tensor Research Articles

Zero-shot reconstruction of ocean sound speed field tensors: A deep plug-and-play approach.

Reconstructing a three-dimensional ocean sound speed field (SSF) from limited and noisy measurements presents an ill-posed and challenging inverse problem. Existing methods used a number of pre-specified priors (e.g., low-rank tensor and tensor neural network structures) to address this issue. However, the SSFs are often too complex to be accurately described by these pre-defined priors. While utilizing neural network-based priors trained on historical SSF data may be a viable workaround, acquiring SSF data remains a nontrivial task. This work starts with a key observation: Although natural images and SSFs admit fairly different characteristics, their denoising processes appear to share similar traits-as both remove random components from more structured signals. This observation allows us to incorporate deep denoisers trained using extensive natural images to realize zero-shot SSF reconstruction, without any extra training or network modifications. To implement this idea, an alternating direction method of multipliers (ADMM) algorithm using such a deep denoiser is proposed, which is reminiscent of the plug-and-play scheme from medical imaging. Our plug-and-play framework is tailored for SSF recovery such that the learned denoiser can be simultaneously used with other handcrafted SSF priors. Extensive numerical studies show that the new framework largely outperforms state-of-the-art baselines, especially under widely recognized challenging scenarios, e.g., when the SSF samples are taken as tensor fibers. The code is available at https://github.com/OceanSTARLab/DeepPnP.

Multidimensional Data Processing With Bayesian Inference via Structural Block Decomposition.

How to handle large multidimensional datasets, such as hyperspectral images and video information, efficiently and effectively plays a critical role in big-data processing. The characteristics of low-rank tensor decomposition in recent years demonstrate the essentials in describing the tensor rank, which often leads to promising approaches. However, most current tensor decomposition models consider the rank-1 component simply to be the vector outer product, which may not fully capture the correlated spatial information effectively for large-scale and high-order multidimensional datasets. In this article, we develop a new novel tensor decomposition model by extending it to the matrix outer product or called Bhattacharya-Mesner product, to form an effective dataset decomposition. The fundamental idea is to decompose tensors structurally in a compact manner as much as possible while retaining data spatial characteristics in a tractable way. By incorporating the framework of the Bayesian inference, a new tensor decomposition model on the subtle matrix unfolding outer product is established for both tensor completion and robust principal component analysis problems, including hyperspectral image completion and denoising, traffic data imputation, and video background subtraction. Numerical experiments on real-world datasets demonstrate the highly desirable effectiveness of the proposed approach.

Automated individual cortical parcellation via consensus graph representation learning

Cortical parcellation plays a pivotal role in elucidating the brain organization. Despite the growing efforts to develop parcellation algorithms using functional magnetic resonance imaging, achieving a balance between intra-individual specificity and inter-individual consistency proves challenging, making the generation of high-quality, subject-consistent cortical parcellations particularly elusive. To solve this problem, our paper proposes a fully automated individual cortical parcellation method based on consensus graph representation learning. The method integrates spectral embedding with low-rank tensor learning into a unified optimization model, which uses group-common connectivity patterns captured by low-rank tensor learning to optimize subjects’ functional networks. This not only ensures consistency in brain representations across different subjects but also enhances the quality of each subject’s representation matrix by eliminating spurious connections. More importantly, it achieves an adaptive balance between intra-individual specificity and inter-individual consistency during this process. Experiments conducted on a test–retest dataset from the Human Connectome Project (HCP) demonstrate that our method outperforms existing methods in terms of reproducibility, functional homogeneity, and alignment with task activation. Extensive network-based comparisons on the HCP S900 dataset reveal that the functional network derived from our cortical parcellation method exhibits greater capabilities in gender identification and behavior prediction than other approaches.

Low-rank tensor learning with projection distance metric for multi-view clustering

Low-rank tensor learning with projection distance metric for multi-view clustering

Enhanced Low-Rank Tensor Recovery Fusing Reweighted Tensor Correlated Total Variation Regularization for Image Denoising

Enhanced Low-Rank Tensor Recovery Fusing Reweighted Tensor Correlated Total Variation Regularization for Image Denoising

Accelerated multi-b-value multi-shot diffusion-weighted imaging based on EPI with keyhole and a low-rank tensor constraint

PurposeMulti-Shot (MS) Echo-Planar Imaging (EPI) may improve the in-plane resolution of multi-b-value DWI, yet it also considerably increases the scan time. Here we explored the combination of EPI with Keyhole (EPIK) and a calibrationless reconstruction algorithm for acceleration of multi-b-value MS-DWI. MethodsWe firstly analyzed the impact of nonuniform phase accrual in EPIK on the reconstructed image. Based on insights gained from the analysis, we developed a calibrationless reconstruction algorithm based on a Space-Contrast-Coil Locally Low-Rank Tensor (SCC-LLRT) constraint for reconstruction of EPIK-acquired data. We compared the algorithm with a modified SPatial-Angular Locally Low-Rank (SPA-LLR) algorithm through simulations, phantoms, and in vivo study. We then compared EPIK with uniformly undersampled EPI for accelerating multi-b-value DWI in 6 healthy subjects. ResultsThrough theoretical derivations, we found that the reconstruction of EPIK with a SENSE-encoding-based algorithm, such as SPA-LLR, may cause additional aliasing artifacts due to the frequency-dependent distortion of the coil sensitivity. Results from simulations, phantoms, and in vivo study verified the theoretical finding by showing that the calibrationless SCC-LLRT algorithm reduced aliasing artifacts compared with SPA-LLR. Finally, EPIK with SCC-LLRT substantially reduced the ghosting artifacts compared with uniform undersampled multi-b-value DWI, decreasing the fitting errors in ADC (0.05 ± 0.01 vs 0.10 ± 0.01, P < 0.001) and IVIM mapping (0.026 ± 0.004 vs 0.06 ± 0.006, P < 0.001). ConclusionThe SCC-LLRT algorithm reduced the aliasing artifacts of EPIK by using a calibrationless modeling of the multi-coil data. The dense sampling of k-space center offers EPIK a potential to improve image quality for acceleration of multi-b-value MS-DWI.

Cluster structure augmented deep nonnegative matrix factorization with low-rank tensor learning

Clustering approaches based on deep nonnegative matrix factorization have received significant attention because it can learn hierarchical semantics from data in a layer-wise manner. However, latent representation learning and clustering are two separate processes in these models, leading to information loss and sub-optimal clusterings. Furthermore, if data contain complex structure or noise, the pre-defined graph in data space will not be precise enough for graph-regularized deep nonnegative matrix factorization models. Moreover, most of them maintain the similarity graph at the top layer, neglecting the information covered by other layers. In this paper, Cluster Structure Augmented Deep Nonnegative Matrix Factorization with Low-rank Tensor Learning is proposed. First, as representations at different layers cover different data abstractions, a learning mechanism is leveraged to generate multiple local similarity graphs based on representations from different layers, thus maintaining diversity across layers. Then, a low-rank third-order tensor is constructed by stacking these graphs to capture the high-order consistency across layers, thus capturing the intrinsic property of data. Third, clustering is integrated into the framework, allowing the cluster structure to facilitate multi-layer matrix factorization and graph structure learning, thereby generating discriminative representations. Experimental results on several datasets demonstrate that the proposed model outperforms several state-of-the-art methods.

Saliency-Guided Sparse Low-Rank Tensor Approximation for Unsupervised Anomaly Detection of Hyperspectral Remote Sensing Images

Hyperspectral anomaly detection can separate sparse anomalies from the low-rank background component under an unsupervised behavior due to sufficient spectral information. Therefore, hyperspectral image anomaly detection technology has great application potential and value in public security and national defense. Currently, most existing models attempt to detect anomalous targets with a sparsity prior, without further considering the visual saliency of the targets themselves. To tackle this issue, this paper proposes a saliency-guided sparse low-rank tensor approximation model, called SSLR, to detect anomalous targets from hyperspectral remote sensing images in an unsupervised manner. Specifically, we first explore the saliency information of each pixel for regularizing the sparse anomaly matrix. We then suggest a three-directional tensor nuclear norm to obtain a low-rank background to characterize the background component. We solve the SSLR optimization problem by an efficient alternating direction method of multipliers framework. Experiments conducted on benchmark hyperspectral datasets demonstrate that the proposed SSLR outperforms some state-of-the-art anomaly detection methods.

Towards Super Compressed Neural Networks for Object Identification: Quantized Low-Rank Tensor Decomposition with Self-Attention

Deep convolutional neural networks have a large number of parameters and require a significant number of floating-point operations during computation, which limits their deployment in situations where the storage space is limited and computational resources are insufficient, such as in mobile phones and small robots. Many network compression methods have been proposed to address the aforementioned issues, including pruning, low-rank decomposition, quantization, etc. However, these methods typically fail to achieve a significant compression ratio in terms of the parameter count. Even when high compression rates are achieved, the network’s performance is often significantly deteriorated, making it difficult to perform tasks effectively. In this study, we propose a more compact representation for neural networks, named Quantized Low-Rank Tensor Decomposition (QLTD), to super compress deep convolutional neural networks. Firstly, we employed low-rank Tucker decomposition to compress the pre-trained weights. Subsequently, to further exploit redundancies within the core tensor and factor matrices obtained through Tucker decomposition, we employed vector quantization to partition and cluster the weights. Simultaneously, we introduced a self-attention module for each core tensor and factor matrix to enhance the training responsiveness in critical regions. The object identification results in the CIFAR10 experiment showed that QLTD achieved a compression ratio of 35.43×, with less than 1% loss in accuracy and a compression ratio of 90.61×, with less than a 2% loss in accuracy. QLTD was able to achieve a significant compression ratio in terms of the parameter count and realize a good balance between compressing parameters and maintaining identification accuracy.

Robust low tubal rank tensor recovery using discrete empirical interpolation method with optimized slice/feature selection

Robust low tubal rank tensor recovery using discrete empirical interpolation method with optimized slice/feature selection

Enforced Block Diagonal Graph Learning for Multikernel Clustering

The existing multikernel graph clustering (MKGC) methods have emerged with notable success on nonlinear clustering tasks since the graph learning can effectively capture graph structure similarity between sample pair. Ideally, a high-quality graph should enjoy the good block diagonal property, i.e. the intercluster similarities correspond to zeros, while intracluster similarities represent nonzeros. Meanwhile, the number of diagonal blocks for a graph equals to the number of clusters on the dataset. However, most of the existing MKGC methods design a corpulent three-parts graph leaning process that poses challenges for hyperparameter tuning, time cost, and clustering performance. To overcome these challenging issues, we propose an enforced block diagonal graph learning for multikernel clustering (EBDGL-MKC) method, where we pursue a high-quality block diagonal graph via well-designed one-part graph leaning scheme rather than three parts. Inspired by symmetric matrix factorization (SMF), we first design a one-part block diagonal graph learning scheme to learn multiple block diagonal graphs, by exploring an explicit theoretical connection between the clustering partition of kernel <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$k$</tex-math> </inline-formula> -means and the excellent block diagonal graph. Then, these block diagonal graphs are stacked into a low-rank tensor for exploiting the high-order structure information hidden in the nonlinear data. After that, an effective alternate algorithm with convergence proof is performed on extensive experiments to demonstrate the superiority of EBDGL compared with the state-of-the-art multikernel clustering (MKC) methods.

Tensor robust PCA with nonconvex and nonlocal regularization

Tensor robust principal component analysis (TRPCA) is a classical way for low-rank tensor recovery, which minimizes the convex surrogate of tensor rank by shrinking each tensor singular value equally. However, for real-world visual data, large singular values represent more significant information than small singular values. In this paper, we propose a nonconvex TRPCA (N-TRPCA) model based on the tensor adjustable logarithmic norm. Unlike TRPCA, our N-TRPCA can adaptively shrink small singular values more and shrink large singular values less. In addition, TRPCA assumes that the whole data tensor is of low rank. This assumption is hardly satisfied in practice for natural visual data, restricting the capability of TRPCA to recover the edges and texture details from noisy images and videos. To this end, we integrate nonlocal self-similarity into N-TRPCA, and further develop a nonconvex and nonlocal TRPCA (NN-TRPCA) model. Specifically, similar nonlocal patches are grouped as a tensor and then each group tensor is recovered by our N-TRPCA. Since the patches in one group are highly correlated, all group tensors have strong low-rank property, leading to an improvement of recovery performance. Experimental results demonstrate that the proposed NN-TRPCA outperforms existing TRPCA methods in visual data recovery. The demo code is available at https://github.com/qguo2010/NN-TRPCA.

Low-Rank Kernel Tensor Learning for Incomplete Multi-View Clustering

Incomplete Multiple Kernel Clustering algorithms, which aim to learn a common latent representation from pre-constructed incomplete multiple kernels from the original data, followed by k-means for clustering. They have attracted intensive attention due to their high computational efficiency. However, our observation reveals that the imputation of these approaches for each kernel ignores the influence of other incomplete kernels. In light of this, we present a novel method called Low-Rank Kernel Tensor Learning for Incomplete Multiple Views Clustering (LRKT-IMVC) to address the above issue. Specifically, LRKT-IMVC first introduces the concept of kernel tensor to explore the inter-view correlations, and then the low-rank kernel tensor constraint is used to further capture the consistency information to impute missing kernel elements, thereby improving the quality of clustering. Moreover, we carefully design an alternative optimization method with promising convergence to solve the resulting optimization problem. The proposed method is compared with recent advances in experiments with different missing ratios on seven well-known datasets, demonstrating its effectiveness and the advantages of the proposed interpolation method.

Towards Multi-Mode Outlier Robust Tensor Ring Decomposition

Conventional Outlier Robust Tensor Decomposition (ORTD) approaches generally represent sparse outlier corruption within a specific mode. However, such an assumption, which may hold for matrices, proves inadequate when applied to high-order tensors. In the tensor domain, the outliers are prone to be corrupted in multiple modes simultaneously. Addressing this limitation, this study proposes a novel ORTD approach by recovering low-rank tensors contaminated by outliers spanning multiple modes. In particular, we conceptualize outliers within high-order tensors as latent tensor group sparsity by decomposing the corrupted tensor into a sum of multiple latent components, where each latent component is exclusive to outliers within a particular direction. Thus, it can effectively mitigate the outlier corruptions prevalent in high-order tensors across multiple modes. To theoretically guarantee recovery performance, we rigorously analyze a non-asymptotic upper bound of the estimation error for the proposed ORTD approach. In the optimization process, we develop an efficient alternate direction method of multipliers (ADMM) algorithm. Empirical validation of the approach's efficacy is undertaken through comprehensive experimentation.

Improving Robustness for Joint Optimization of Camera Pose and Decomposed Low-Rank Tensorial Radiance Fields

In this paper, we propose an algorithm that allows joint refinement of camera pose and scene geometry represented by decomposed low-rank tensor, using only 2D images as supervision. First, we conduct a pilot study based on a 1D signal and relate our findings to 3D scenarios, where the naive joint pose optimization on voxel-based NeRFs can easily lead to sub-optimal solutions. Moreover, based on the analysis of the frequency spectrum, we propose to apply convolutional Gaussian filters on 2D and 3D radiance fields for a coarse-to-fine training schedule that enables joint camera pose optimization. Leveraging the decomposition property in decomposed low-rank tensor, our method achieves an equivalent effect to brute-force 3D convolution with only incurring little computational overhead. To further improve the robustness and stability of joint optimization, we also propose techniques of smoothed 2D supervision, randomly scaled kernel parameters, and edge-guided loss mask. Extensive quantitative and qualitative evaluations demonstrate that our proposed framework achieves superior performance in novel view synthesis as well as rapid convergence for optimization. The source code is available at https://github.com/Nemo1999/Joint-TensoRF.

Randomized two-sided subspace iteration for low-rank matrix and tensor decomposition

The low-rank approximation of big data matrices and tensors plays a pivotal role in many modern applications. Although, a truncated version of the singular value decomposition (SVD) furnishes the best approximation, its computation is challenging on modern, multicore architectures. Recently, the randomized subspace iteration has shown to be a powerful tool in approximating large-scale matrices. In this paper we present a two-sided variant of the randomized subspace iteration. Novel in our work is the exploitation of the unpivoted QR factorization, rather than the SVD, for factorizing the compressed matrix. Hence our algorithm is a randomized rank-revealing URV decomposition. We prove the rank-revealingness of our algorithm by establishing bounds for the singular values as well as the other blocks of the compressed matrix. We further provide bounds on the error of the low-rank approximations of the proposed algorithm, in both 2- and Frobenius norm. In addition, we employ the proposed algorithm to efficiently compute low rank tensor decompositions: we present two randomized algorithms, one for the truncated higher-order SVD, and the other for the tensor SVD. We conduct numerical tests on (i) various classes of matrices, and (ii) synthetic tensors and real datasets to demonstrate the efficacy of the proposed algorithms.

Error-robust multi-view subspace clustering with nonconvex low-rank tensor approximation and hyper-Laplacian graph embedding

Tensor based subspace clustering, a method aimed at partitioning multi-view data into distinct clusters through the tensor low-rank representation, has gained widespread popularity. Despite the effectiveness of tensor singular value decomposition (t-SVD) based nuclear norm in extracting high-dimensional information from multiple views, certain limitations persist. Firstly, while the commonly used l2,1 norm enhances clustering robustness, it remains susceptible to the impact of high-intensity noises. Secondly, the t-SVD based nuclear norm provides a biased estimation of tensor rank. Thirdly, existing methods either fail to preserve manifold structures from the original space or only extract binary relationships between data points. To address these challenges and enhance clustering performance in both high-intensity noisy and real-world conditions, an error-robust multi-view subspace clustering with nonconvex low-rank tensor approximation and hyper-Laplacian graph embedding (EMSC-NLTHG) method is proposed. Specifically, Cauchy loss function which reduces sensitivity to larger noises and outliers is introduced. Based on the Cauchy loss function, we develop the Cauchy pseudo norm to represent the reconstruction error in clustering to enhance the robustness. Moreover, rather than uniformly weighting all singular values in the t-SVD nuclear norm, a nonconvex tensor nuclear norm is adopted to add weights adaptively and approximate the true tensor rank. Additionally, hypergraph is embedded in the representation matrices to preserve local manifold structures and extract the complex multivariate relationships between data points. Extensive experiments demonstrate that the proposed EMSC-NLTHG method consistently outperforms state-of-the-art techniques by approximately 18% to 40% on datasets corrupted by high-intensity noises and around 1% to 27% on eight real-world datasets across six popular evaluation metrics.

Phase retrieval from integrated intensity of auto-convolution

Ultra-fast optical pulses are the most ephemeral sensing paradigm ever devised, examining events over incredibly brief timescales with broadband illumination. A consequence of sensing at timescales lower than a picosecond is that pulse characterization cannot be done with traditional analog-to-digital samplers and must be ascertained from integrating intensity sensors. Techniques for pulse characterization have been constructed using combinations of time-invariant and time-variant filter responses to create non-linear but invertible intensity datasets (Walmsley & Dorrer, 2009). In this paper, we develop a novel high-order phase retrieval technique to perform pulse characterization from a single-pixel integrating sensor measuring integrated intensity of auto-convolution (IIAC). We examine gradient descent’s ability to recover signals as a function of signal dimension and measurement count, and we demonstrate the effective use of iterative hard tensor thresholding as an initializer. Finally, we demonstrate IIAC recovery in a laboratory setting to recover the time profile of a complex laser pulse. We assert that the IIAC recovery solution demonstrated here simultaneously provides the optics community with a pulse characterization technique that scales to low-power microscopy systems and provides the optimization community with a physically motivated high-order phase retrieval problem enhanced by low-rank tensor processing.

Stochastic Low-Rank Tensor Bandits for Multi-Dimensional Online Decision Making

Multi-dimensional online decision making plays a crucial role in many real applications such as online recommendation and digital marketing. In these problems, a decision at each time is a combination of choices from different types of entities. To solve it, we introduce stochastic low-rank tensor bandits, a class of bandits whose mean rewards can be represented as a low-rank tensor. We consider two settings, tensor bandits without context and tensor bandits with context. In the first setting, the platform aims to find the optimal decision with the highest expected reward, a.k.a, the largest entry of true reward tensor. In the second setting, some modes of the tensor are contexts and the rest modes are decisions, and the goal is to find the optimal decision given the contextual information. We propose two learning algorithms tensor elimination and tensor epoch-greedy for tensor bandits without context, and derive finite-time regret bounds for them. Comparing with existing competitive methods, tensor elimination has the best overall regret bound and tensor epoch-greedy has a sharper dependency on dimensions of the reward tensor. Furthermore, we develop a practically effective Bayesian algorithm called tensor ensemble sampling for tensor bandits with context. Extensive simulations and real analysis in online advertising data back up our theoretical findings and show that our algorithms outperform various state-of-the-art approaches that ignore the tensor low-rank structure. Supplementary materials for this article are available online including a standardized description of the materials available for reproducing the work.

The effect of smooth parametrizations on nonconvex optimization landscapes

AbstractWe develop new tools to study landscapes in nonconvex optimization. Given one optimization problem, we pair it with another by smoothly parametrizing the domain. This is either for practical purposes (e.g., to use smooth optimization algorithms with good guarantees) or for theoretical purposes (e.g., to reveal that the landscape satisfies a strict saddle property). In both cases, the central question is: how do the landscapes of the two problems relate? More precisely: how do desirable points such as local minima and critical points in one problem relate to those in the other problem? A key finding in this paper is that these relations are often determined by the parametrization itself, and are almost entirely independent of the cost function. Accordingly, we introduce a general framework to study parametrizations by their effect on landscapes. The framework enables us to obtain new guarantees for an array of problems, some of which were previously treated on a case-by-case basis in the literature. Applications include: optimizing low-rank matrices and tensors through factorizations; solving semidefinite programs via the Burer–Monteiro approach; training neural networks by optimizing their weights and biases; and quotienting out symmetries.