Sum Of Kronecker Products Research Articles

Parameterized hypercomplex convolutional network for accurate protein backbone torsion angle prediction

Predicting the backbone torsion angles corresponding to each residue of a protein from its amino acid sequence alone is a challenging problem in computational biology. Existing torsion angle predictors mainly use profile features, which are generated by performing time-consuming multiple sequence alignments, for torsion angle prediction. Compared with traditional profile features, embedding features from pretrained protein language models have significant advantages in prediction performance and computational speed. However, embedding features usually have higher dimensions and different embedding features have significantly different dimensions. To this end, we design a novel parameter-efficient deep torsion angle predictor, PHAngle, specifically for embedding features. PHAngle is a parameterized hypercomplex convolutional network consisting of parameterized hypercomplex linear and convolutional layers whose weight parameters can be characterized as the sum of Kronecker products. Experimental results on six benchmark test sets including TEST2016, TEST2018, TEST2020_HQ, CASP12, CASP13 and CASP-FM demonstrate that PHAngle achieves the state-of-the-art torsion angle performance with the fewest parameters compared to the nine existing methods. The source code and datasets are available at https://github.com/fengtuan/PHAngle.

Structured Matrix Approximations via Tensor Decompositions

We provide a computational framework for approximating a class of structured matrices; here, the term structure is very general, and may refer to a regular sparsity pattern (e.g., block-banded), or be more highly structured (e.g., symmetric block Toeplitz). The goal is to uncover {\it additional latent structure} that will in turn lead to computationally efficient algorithms when the new structured matrix approximations are employed in the place of the original operator. Our approach has three steps: map the structured matrix to tensors, use tensor compression algorithms, and map the compressed tensors back to obtain two different matrix representations -- sum of Kronecker products and block low-rank format. The use of tensor decompositions enables us to uncover latent structure in the problem and leads to compressed representations of the original matrix that can be used efficiently in applications. The resulting matrix approximations are memory efficient, easy to compute with, and preserve the error that is due to the tensor compression in the Frobenius norm. Our framework is quite general. We illustrate the ability of our method to uncover block-low-rank format on structured matrices from two applications: system identification, space-time covariance matrices. In addition, we demonstrate that our approach can uncover sum of structured Kronecker products structure on several matrices from the SuiteSparse collection. Finally, we show that our framework is broad enough to encompass and improve on other related results from the literature, as we illustrate with the approximation of a three-dimensional blurring operator.

Generalized vec trick for fast learning of pairwise kernel models

Pairwise learning corresponds to the supervised learning setting where the goal is to make predictions for pairs of objects. Prominent applications include predicting drug-target or protein-protein interactions, or customer-product preferences. In this work, we present a comprehensive review of pairwise kernels, that have been proposed for incorporating prior knowledge about the relationship between the objects. Specifically, we consider the standard, symmetric and anti-symmetric Kronecker product kernels, metric-learning, Cartesian, ranking, as well as linear, polynomial and Gaussian kernels. Recently, a O(nm+nq) time generalized vec trick algorithm, where n, m, and q denote the number of pairs, drugs and targets, was introduced for training kernel methods with the Kronecker product kernel. This was a significant improvement over previous O(n^2) training methods, since in most real-world applications m,q<< n. In this work we show how all the reviewed kernels can be expressed as sums of Kronecker products, allowing the use of generalized vec trick for speeding up their computation. In the experiments, we demonstrate how the introduced approach allows scaling pairwise kernels to much larger data sets than previously feasible, and provide an extensive comparison of the kernels on a number of biological interaction prediction tasks.

Forest Height and Underlying Topography Inversion Using Polarimetric SAR Tomography Based on SKP Decomposition and Maximum Likelihood Estimation

The key point of forest height and underlying topography inversion using synthetic aperture radar tomography (TomoSAR) depends on the accurate positioning of the phase centers of different scattering mechanisms. The traditional nonparametric spectrum analysis methods (such as beamforming and Capon) have limited vertical resolution and cannot accurately distinguish closely spaced scatterers. In addition, it is very difficult to accurately estimate the ground or canopy heights with single polarimetric SAR images because there is no guarantee that the vertical profile will generate two clear and separate peaks for all resolution cells. A polarimetric TomoSAR method based on SKP (sum of Kronecker products) decomposition and iterative maximum likelihood estimation is proposed in this paper. On the one hand, the iterative maximum likelihood TomoSAR method has a higher vertical resolution than that of the traditional methods. On the other hand, the separation of the canopy scattering mechanism and the ground scattering mechanism is conducive to the positioning of the phase centers. This method was applied to the inversion of forest height and underlying topography in a tropical forest over the TropiSAR2009 test site in Paracou, French Guiana with six passes of polarimetric SAR images. The inversion accuracy of underlying topography of the proposed method was up to 1.489 m and the inversion accuracy of forest height was up to 1.765 m. Compared with the traditional polarimetric beamforming and polarimetric capon methods, the proposed method greatly improved the inversion accuracy of forest height and underlying topography.

Sum of Kronecker products representation and its Cholesky factorization for spatial covariance matrices from large grids

The sum of Kronecker products (SKP) representation for spatial covariance matrices from gridded observations and a corresponding adaptive-cross-approximation-based framework for building the Kronecker factors are investigated. The time cost for constructing an n-dimensional covariance matrix is O(nk2) and the total memory footprint is O(nk), where k is the number of Kronecker factors. The memory footprint under the SKP representation is compared with that under the hierarchical representation and found to be one order of magnitude smaller. A Cholesky factorization algorithm under the SKP representation is proposed and shown to factorize a one-million dimensional covariance matrix in under 600 seconds on a standard scientific workstation. With the computed Cholesky factor, simulations of Gaussian random fields in one million dimensions can be achieved at a low cost for a wide range of spatial covariance functions.

The Extended Manifold for Antenna Arrays

The paper investigates a general parametric model for the manifold of polarization sensitive arrays based on representing each antenna by a collection of short dipoles. The manifold has a simple form consisting of an extended isotropic manifold, vertical and horizontal direction vectors and a constant non-square coupling matrix. The resulting manifold captures the effects of coupling, polarization and antenna characteristics. It provides an accurate representation of the analytic manifold which is based on electromagnetic theory. Three methods for computing the Coupling Matrix are described based on: (i) the current distributions in the antennas, (ii) its approximation by a sum of Kronecker products, and (iii) least-squares fit of the analytic and extended manifolds. The accuracy of these manifolds is illustrated by numerical examples.

Recursive Kronecker-Based Vector Autoregressive Identification for Large-Scale Adaptive Optics

This brief presents an algorithm for the recursive identification of Vector AutoRegressive (VAR) models of large dimensions. We consider a VAR model where the coefficient matrices can be written as a sum of Kronecker products. The algorithm proposed consists of recursively updating the Kronecker factor matrices at each new time step using alternating least squares. When the number of terms in the Kronecker sum is small, a significant reduction in computational complexity is achieved with respect to the recursive least squares algorithm on an unstructured VAR model. Numerical validation of nonstationary atmospheric turbulence data, both synthetic and experimental, is shown for an adaptive optics application. Significant improvements in accuracy over batch identification methods that assume stationarity are observed while both the computational complexity and the required storage are reduced.

TomoSAR Imaging for the Study of Forested Areas: A Virtual Adaptive Beamforming Approach

Among the objectives of the upcoming space missions Tandem-L and BIOMASS, is the 3-D representation of the global forest structure via synthetic aperture radar (SAR) tomography (TomoSAR). To achieve such a goal, modern approaches suggest solving the TomoSAR inverse problems by exploiting polarimetric diversity and structural model properties of the different scattering mechanisms. This way, the related tomographic imaging problems are treated in descriptive regularization settings, applying modern non-parametric spatial spectral analysis (SSA) techniques. Nonetheless, the achievable resolution of the commonly performed SSA-based estimators highly depends on the span of the tomographic aperture; furthermore, irregular sampling and non-uniform constellations sacrifice the attainable resolution, introduce artifacts and increase ambiguity. Overcoming these drawbacks, in this paper, we address a new multi-stage iterative technique for feature-enhanced TomoSAR imaging that aggregates the virtual adaptive beamforming (VAB)-based SSA approach, with the wavelet domain thresholding (WDT) regularization framework, which we refer to as WAVAB (WDT-refined VAB). First, high resolution imagery is recovered applying the descriptive experiment design regularization (DEDR)-inspired reconstructive processing. Next, the additional resolution enhancement with suppression of artifacts is performed, via the WDT-based sparsity promoting refinement in the wavelet transform (WT) domain. Additionally, incorporation of the sum of Kronecker products (SKP) decomposition technique at the pre-processing stage, improves ground and canopy separation and allows for the utilization of different better adapted TomoSAR imaging techniques, on the ground and canopy structural components, separately. The feature enhancing capabilities of the novel robust WAVAB TomoSAR imaging technique are corroborated through the processing of airborne data of the German Aerospace Center (DLR), providing detailed volume height profiles reconstruction, as an alternative to the competing non-parametric SSA-based methods.

Tensor-based predictive control for extremely large-scale single conjugate adaptive optics.

In this paper we propose a data-driven predictive control algorithm for large-scale single conjugate adaptive optics systems. At each time sample, the Shack-Hartmann wavefront sensor signal sampled on a spatial grid of size N×N is reshuffled into a d-dimensional tensor. Its spatial-temporal dynamics are modeled with a d-dimensional autoregressive model of temporal order p, where each tensor storing past data undergoes a multilinear transformation by factor matrices of small sizes. Equivalently, the vector form of this autoregressive model features coefficient matrices parametrized with a sum of Kronecker products between d-factor matrices. We propose an Alternating Least Squares algorithm for identifying the factor matrices from open-loop sensor data. When modeling each coefficient matrix with a sum of r terms, the computational complexity for updating the sensor prediction online reduces from O(pN4) in the unstructured matrix case to O(prd N2(d+1)d). Most importantly, this model structure breaks away from assuming any prior spatial-temporal coupling as it is discovered from the data. The algorithm is validated on a laboratory testbed that demonstrates the ability to accurately decompose the coefficient matrices of large-scale autoregressive models with a tensor-based representation, hence achieving high data compression rates and reducing the temporal error especially for a large Greenwood per sample frequency ratio.

Model-Based Target Scattering Decomposition of Polarimetric SAR Tomography

When dealing with forest scenario, target scattering separation using synthetic aperture radar (SAR) tomography is a challenging task for the application of biophysical parameter retrieval approaches. One important and widely popular solution used to investigate the scattering mechanism separation based on multipolarimetric multibaseline (MPMB) SAR data is the sum of Kronecker products (SKPs), which provides the basis for decomposition of the data into ground-only and canopy-only contributions. In this paper, we investigate the possibility of characterizing multiple scattering mechanisms using the SKPs of covariance matrix. In particular, we present a method for characterization of forest structure using MPMB data that adapt SKP with the generalized volume description and the physical model of interferometric cross correlation as the sum of scattering contributions. According to the Freeman–Durden model, the method expresses the estimated covariance matrix in terms of the Kronecker product of polarimetric and interferometric coherence matrices corresponding to direct, double-bounce, and random-volume scattering mechanisms. The proposed method is tested with simulated and P-band MB data acquired by ONERA over a tropical forest in French Guiana in the frame of the European Space Agency’s campaign TROPISAR. Comparison of the retrieved height of trees with a LiDAR-based canopy model as a reference showed that the proposed method has the potential to decrease root-mean-square error of forest height by up to 3.9 m with respect to SKP.

Singular Value Decomposition Approximation via Kronecker Summations for Imaging Applications

In this paper we propose an approach to approximate a truncated singular value decomposition of a large structured matrix. By first decomposing the matrix into a sum of Kronecker products, our approach can be used to approximate a large number of singular values and vectors more efficiently than other well-known schemes, such as iterative algorithms based on Golub--Kahan--Lanczos bidiagonalization. We provide theoretical results and numerical experiments to demonstrate the accuracy of our approximation and show how the approximation can be used to solve large scale ill-posed inverse problems, either as an approximate filtering method, or as a preconditioner to accelerate iterative algorithms.

QUARKS: Identification of Large-Scale Kronecker Vector-AutoRegressive Models

In this paper, we address the identification of two-dimensional (2-D) spatial-temporal dynamical systems described by the Vector-AutoRegressive (VAR) form. The coefficient-matrices of the VAR model are parametrized as sums of Kronecker products. When the number of terms in the sum is small compared to the size of the matrices, such a Kronecker representation efficiently models large-scale VAR models. Estimating the coefficient matrices in least-squares sense gives rise to a bilinear estimation problem that is tackled using an Alternating Least Squares (ALS) algorithm. Regularization or parameter constraints on the coefficient-matrices allows to induce temporal network properties, such as stability, as well as spatial properties, such as sparsity or Toeplitz structure. Convergence of the regularized ALS is proved using fixed-point theory. A numerical example demonstrates the advantages of the new modeling paradigm. It leads to comparable variance of the prediction error with the unstructured least-squares estimation of VAR models. However, the number of parameters grows only linearly with respect to the number of nodes in the 2-D sensor network instead of quadratically in the case of fully unstructured coefficient-matrices.

Reordering for improving global Arnoldi–Tikhonov method in image restoration problems

This paper discusses the solution of large-scale linear discrete ill-posed problems arising from image restoration problems. Since the scale of the problem is usually very large, the computations with the blurring matrix can be very expensive. In this regard, we consider problems in which the coefficient matrix is the sum of Kronecker products of matrices to benefit the computation. Here, we present an alternative approach based on reordering of the image approximations obtained with the global Arnoldi–Tikhonov method. The ordering of the intensities is such that it makes the image approximation monotonic and thus minimizes the finite differences norm. We present theoretical properties of the method and numerical experiments on image restoration.

Kronecker-ARX models in identifying (2D) spatial-temporal systems

In this paper we address the identification of (2D) spatial-temporal dynamical systems governed by the Vector Auto-Regressive (VAR) form. The coefficient-matrices of the VAR model are parametrized as sums of Kronecker products. When the number of terms in the sum is small compared to the size of the matrix, such a Kronecker representation leads to high data compression. Estimating in least-squares sense the coefficient-matrices gives rise to a bilinear estimation problem, which is tackled using a three-stage algorithm. A numerical example demonstrates the advantages of the new modeling paradigm. It leads to comparable performances with the unstructured least-squares estimation of VAR models. However, the number of parameters in the new modeling paradigm grows linearly w.r.t. the number of nodes in the 2D sensor network instead of quadratically in the full unstructured matrix case.

Learning Dictionaries as a Sum of Kronecker Products

The choice of an appropriate frame, or dictionary, is a crucial step in the sparse representation of a given class of signals. Traditional dictionary learning techniques generally lead to unstructured dictionaries that are costly to deploy and train, and do not scale well to higher dimensional signals. In order to overcome such limitation, we propose a learning algorithm that constrains the dictionary to be a sum of Kronecker products of smaller subdictionaries. This approach, named sum of Kronecker products, is demonstrated experimentally in an image denoising application.

Approximation of functions of large matrices with Kronecker structure

We consider the numerical approximation of $$f(\mathcal{A})b$$f(A)b where $$b\in {\mathbb {R}}^{N}$$bźRN and $$\mathcal A$$A is the sum of Kronecker products, that is $$\mathcal{A}=M_2 \otimes I + I \otimes M_1\in {\mathbb {R}}^{N\times N}$$A=M2źI+IźM1źRN×N. Here f is a regular function such that $$f(\mathcal{A})$$f(A) is well defined. We derive a computational strategy that significantly lowers the memory requirements and computational efforts of the standard approximations, with special emphasis on the exponential and on completely monotonic functions, for which the new procedure becomes particularly advantageous. Our findings are illustrated by numerical experiments with typical functions used in applications.

Spline approximation, Kronecker products and multilinear forms

SummarySums of Kronecker products occur naturally in high‐dimensional spline approximation problems, which arise, for example, in the numerical treatment of chemical reactions. In full matrix form, the resulting non‐sparse linear problems usually exceed the memory capacity of workstations. We present methods for the manipulation and numerical handling of Kronecker products in factorized form. Moreover, we analyze the problem of approximating a given matrix by sums of Kronecker products by making use of the equivalence to the problem of decomposing multilinear forms into sums of one‐forms. Greedy algorithms based on the maximization of multilinear forms over a torus are used to obtain such (finite and infinite) decompositions that can be used to solve the approximation problem. Moreover, we present numerical considerations for these algorithms. Copyright © 2016 John Wiley &amp; Sons, Ltd.

A Framework for Regularization via Operator Approximation

Regularization approaches based on spectral filtering can be highly effective in solving ill-posed inverse problems. These methods, however, require computing the singular value decomposition (SVD) and choosing appropriate regularization parameters. These tasks can be prohibitively expensive for large-scale problems. In this paper, we present a framework that uses operator approximations to efficiently obtain good regularization parameters without an SVD of the original operator. Instead, we approximate the original operator with a nearby structured or separable one whose SVD is easily computable. Highly effective methods can then be used to efficiently compute good regularization parameters for the nearby problem. Then, we solve the original problem iteratively using the regularization determined for the approximate problem. A variety of regularization approaches can be incorporated into this framework, but we focus here on the recently developed windowed regularization, a generalization of Tikhonov regularization in which different regularization parameters are used in different regions of the spectrum. We derive bounds on the perturbation to the computed solution and residual resulting from using the regularization determined for the approximate operator. We demonstrate the effectiveness of our method in computations using operator approximations such as sums of Kronecker products, block circulant with circulant blocks matrices, and Krylov subspace approximations.

On the numerical solution of Kronecker-based infinite level-dependent QBD processes

Infinite level-dependent quasi-birth-and-death (LDQBD) processes can be used to model Markovian systems with countably infinite multidimensional state spaces. Recently it has been shown that sums of Kronecker products can be used to represent the nonzero blocks of the transition rate matrix underlying an LDQBD process for models from stochastic chemical kinetics. This paper extends the form of the transition rates used recently so that a larger class of models including those of call centers can be analyzed for their steady-state. The challenge in the matrix analytic solution then is to compute conditional expected sojourn time matrices of the LDQBD model under low memory and time requirements after truncating its countably infinite state space judiciously. Results of numerical experiments are presented using a Kronecker-based matrix-analytic solution on models with two or more countably infinite dimensions and rules of thumb regarding better implementations are derived. In doing this, a more recent approach that reduces memory requirements further by enabling the computation of steady-state expectations without having to obtain the steady-state distribution is also considered.

Kronecker-Based Infinite Level-Dependent QBD Processes

Markovian systems with multiple interacting subsystems under the influence of a control unit are considered. The state spaces of the subsystems are countably infinite, whereas that of the control unit is finite. A recent infinite level-dependent quasi-birth-and-death model for such systems is extended by facilitating the automatic representation and generation of the nonzero blocks in its underlying infinitesimal generator matrix with sums of Kronecker products. Experiments are performed on systems of stochastic chemical kinetics having two or more countably infinite state space subsystems. Results indicate that, even though more memory is consumed, there are many cases where a matrix-analytic solution coupled with Lyapunov theory yields a faster and more accurate steady-state measure compared to that obtained with simulation.