Tensor Kernels Research Articles

Low-Rank Kernel Tensor Learning for Incomplete Multi-View Clustering

Incomplete Multiple Kernel Clustering algorithms, which aim to learn a common latent representation from pre-constructed incomplete multiple kernels from the original data, followed by k-means for clustering. They have attracted intensive attention due to their high computational efficiency. However, our observation reveals that the imputation of these approaches for each kernel ignores the influence of other incomplete kernels. In light of this, we present a novel method called Low-Rank Kernel Tensor Learning for Incomplete Multiple Views Clustering (LRKT-IMVC) to address the above issue. Specifically, LRKT-IMVC first introduces the concept of kernel tensor to explore the inter-view correlations, and then the low-rank kernel tensor constraint is used to further capture the consistency information to impute missing kernel elements, thereby improving the quality of clustering. Moreover, we carefully design an alternative optimization method with promising convergence to solve the resulting optimization problem. The proposed method is compared with recent advances in experiments with different missing ratios on seven well-known datasets, demonstrating its effectiveness and the advantages of the proposed interpolation method.

Learning Kernel-Modulated Neural Representation for Efficient Light Field Compression.

Light fields capture 3D scene information by recording light rays emitted from a scene at various orientations. They offer a more immersive perception, compared with classic 2D images, but at the cost of huge data volumes. In this paper, we design a compact neural network representation for the light field compression task. In the same vein as the deep image prior, the neural network takes randomly initialized noise as input and is trained in a supervised manner in order to best reconstruct the target light field Sub-Aperture Images (SAIs). The network is composed of two types of complementary kernels: descriptive kernels (descriptors) that store scene description information learned during training, and modulatory kernels (modulators) that control the rendering of different SAIs from the queried perspectives. To further enhance compactness of the network meanwhile retain high quality of the decoded light field, we propose modulator allocation and apply kernel tensor decomposition techniques, followed by non-uniform quantization and lossless entropy coding. Extensive experiments demonstrate that our method outperforms other state-of-the-art (SOTA) methods by a significant margin in the light field compression task. Moreover, after adapting descriptors, the modulators learned from one light field can be transferred to new light fields for rendering dense views, showing the potential of the solution for view synthesis.

Leveraging tensor kernels to reduce objective function mismatch in deep clustering

Objective Function Mismatch (OFM) occurs when the optimization of one objective has a negative impact on the optimization of another objective. In this work we study OFM in deep clustering, and find that the popular autoencoder-based approach to deep clustering can lead to both reduced clustering performance, and a significant amount of OFM between the reconstruction and clustering objectives. To reduce the mismatch, while maintaining the structure-preserving property of an auxiliary objective, we propose a set of new auxiliary objectives for deep clustering, referred to as the Unsupervised Companion Objectives (UCOs). The UCOs rely on a kernel function to formulate a clustering objective on intermediate representations in the network. Generally, intermediate representations can include other dimensions, for instance spatial or temporal, in addition to the feature dimension. We therefore argue that the naïve approach of vectorizing and applying a vector kernel is suboptimal for such representations, as it ignores the information contained in the other dimensions. To address this drawback, we equip the UCOs with structure-exploiting tensor kernels, designed for tensors of arbitrary rank. The UCOs can thus be adapted to a broad class of network architectures. We also propose a novel, regression-based measure of OFM, allowing us to accurately quantify the amount of OFM observed during training. Our experiments show that the OFM between the UCOs and the main clustering objective is lower, compared to a similar autoencoder-based model. Further, we illustrate that the UCOs improve the clustering performance of the model, in contrast to the autoencoder-based approach. The code for our experiments is available at https://github.com/danieltrosten/tk-uco.

Regularity and convergence results for nonlocal peridynamic equations with truncated tensor kernels

Regularity and convergence results for nonlocal peridynamic equations with truncated tensor kernels

Precise crop classification of UAV hyperspectral imagery using kernel tensor slice sparse coding based classifier

Precise crop classification plays a significant role in the agriculture field. An appropriate data source for precise crop classification is high spatial resolutions hyperspectral imagery (H2 imagery) acquired by unmanned aerial vehicle (UAV). However, for imagery with many different classes of crops, crop classification of UAV H2 imagery is a huge challenge. The significant spectral diversity, spatial heterogeneity and nonlinear data structure of UAV H2 imagery results in poor spectral discriminability. To improve the discriminability, a kernel tensor slice sparse coding-based classifier (KTSSCC) is proposed for precise crop classification of UAV H2 imagery in this research. The kernel tensor representation mechanism in KTSSCC can reduce the nonlinear separation while well preserving the spectral characteristics and spatial constraints of land-covers, and thus the discriminability is greatly improved. Furthermore, this paper puts forward the kernel tensor slice sparse orthogonal matching pursuit (KTSSOMP) algorithm to optimize kernel tensor slice sparse coding in the spectral space, which greatly reduces the computation cost. Moreover, there are very few parameters to be tuned in our proposed model. We assess the performance of KTSSCC on two real UAV hyperspectral imagery datasets, and find that, based on visual and quantitative results, it provides satisfactory crop classification results and outperforms the state-of-the-art approaches.

Auto-weighted multiple kernel tensor clustering

Multiple kernel subspace clustering (MKSC) has attracted intensive attention since its powerful capability of exploring consensus information by generating a high-quality affinity graph from multiple base kernels. However, the existing MKSC methods still exist the following limitations: (1) they essentially neglect the high-order correlations hidden in different base kernels; and (2) they perform candidate affinity graph learning and consensus affinity graph learning in two separate steps, where suboptimal solution may be obtained. To alleviate these problems, a novel MKSC method, namely auto-weighted multiple kernel tensor clustering (AMKTC), is proposed. Specifically, AMKTC first integrates the consensus affinity graph learning and candidate affinity graph learning into a unified framework, where the optimal goal can be achieved by making these two learning processes negotiate with each other. Further, an auto-weighted fusion scheme with one-step manner is proposed to learn the final consensus affinity graph, where the reasonable weights will be automatically learned for each candidate graph. Finally, the essential high-order correlations between multiple base kernels can be captured by leveraging tensor-singular value decomposition (t-SVD)-based tensor nuclear norm constraint on a 3-order graph tensor. Experiments on seven benchmark datasets with eleven comparison methods demonstrate that our method achieves state-of-the-art clustering performance.

Analysis of Deep Convolutional Neural Networks Using Tensor Kernels and Matrix-Based Entropy.

Analyzing deep neural networks (DNNs) via information plane (IP) theory has gained tremendous attention recently to gain insight into, among others, DNNs' generalization ability. However, it is by no means obvious how to estimate the mutual information (MI) between each hidden layer and the input/desired output to construct the IP. For instance, hidden layers with many neurons require MI estimators with robustness toward the high dimensionality associated with such layers. MI estimators should also be able to handle convolutional layers while at the same time being computationally tractable to scale to large networks. Existing IP methods have not been able to study truly deep convolutional neural networks (CNNs). We propose an IP analysis using the new matrix-based Rényi's entropy coupled with tensor kernels, leveraging the power of kernel methods to represent properties of the probability distribution independently of the dimensionality of the data. Our results shed new light on previous studies concerning small-scale DNNs using a completely new approach. We provide a comprehensive IP analysis of large-scale CNNs, investigating the different training phases and providing new insights into the training dynamics of large-scale neural networks.

Kernel Tensor Decomposition can improve the drug discovery process

Kernel Tensor Decomposition can improve the drug discovery process Kernel tensor decomposition and its use in drug discovery for SARS-CoV-2 was vital, however, due to its general method, it has the potential to be used for a wide range of future problems. Through the series of my previous articles (1-5) that target the general audience, I have been avoiding discussing mathematical details that non-expert cannot easily understand. Nevertheless, I discuss these in this article since it is seemingly the last one about my COVD-19 studies. As a researcher of bioinformatics, my main interest is in the mathematical side. Thus avoiding mathematics has already prevented me from explaining well what I am really interested in.

Hybrid Fully Connected Tensorized Compression Network for Hyperspectral Image Classification

Deep learning models, such as convolutional neural networks (CNNs), have made significant progress in hyperspectral image (HSI) classification. However, these models require a large number of parameters, which occupy a lot of storage space and suffer from overfitting, thus resulting in performance loss. To solve the above problems, in this article, we propose a new compression network [namely, a Hybrid Fully Connected Tensorized Compression Network (HybridFCTCN)] by considering the high dimensionality of HSI data. First, using the low-rank fully connected tensor network decomposition (FCTND), three novel units, i.e., FCTN-FC, FCTNConv2D, and FCTNConv3D, are designed to compress the weight tensor of standard fully connected (FC) layer and kernel tensor of convolutional layer, reducing their parameters. In the novel units, the intrinsic correlation of the decomposed factors is adequately exploited by the FC structures, which enhances their feature extraction and classification abilities. Then, benefiting from the hybrid network backbone composed of the FCTNConv3D and FCTNConv2D units, HybridFCTCN can extract more discriminative features with fewer parameters, while it has great generalization capability and robustness, enabling better HSI classification. Finally, the rank of above-designed units is defined, and its determination is discussed to facilitate the application of the proposed model. Extensive experiments on three widely used HSI datasets reveal that the proposed model achieves state-of-the-art classification performance for different training sample sizes with a very small number of parameters.

New regularization methods for convolutional kernel tensors

<abstract><p>Convolution is a very basic and important operation for convolutional neural networks. For neural network training, how to bound the convolutional layers is a currently popular research topic. Each convolutional layer is represented by a tensor, which corresponds to a structured transformation matrix. The objective is to ensure that the singular values of each transformation matrix are bounded around 1 by changing the entries of the tensor. We propose three new regularization terms for a convolutional kernel tensor and derive the gradient descent algorithm for each penalty function. Numerical examples are presented to demonstrate the effectiveness of the algorithms.</p></abstract>

Tensor-Based Underdetermined Blind Identification of Instantaneous Mixtures

In order to explore the effective statistical information of observed signals to stack into a tensor, while reduce the computational complexity and speed up the convergence, a new tensor-based underdetermined blind identification method of instantaneous mixture is proposed in this brief. Firstly, the autocovariance matrix of each observed sub-block is calculated to construct the symmetric third-order tensor according to segmentation scheme. Secondly, the original tensor is reduced into a low-rank kernel tensor by truncated multi-linear singular value decomposition (MLSVD). Finally, an enhanced line search (ELS) technique is applied to speed up the convergence of alternating least squares (ALS) to identify the mixing matrix. The simulation results indicate that the presented approach provides a superior estimated performance compared to the existing methods in both real and complex scenarios.

DOA Estimation via Tensor Truncated Convolution Nuclear Norm Minimization in MIMO Radar Under Array Sensor Failures

Exploring the inherent multidimensional structures has great potential to obtain more latent information for reconstructing the missing data resulting from array sensor failures in multiple-input–multiple-output (MIMO) radar. Most tensor completion (TC) methods based on the minimization of tensor nuclear norms are adopted to fill randomly distributed missing entries. However, the randomness of missing entries is violated in MIMO radar under sensor failures, where some slices are missing entirely. Thus, these existing methods become ineffective when faced with such type of structurally missing entries. To remedy this, we propose an improved TC method based on tensor truncated convolution nuclear norm minimization (CNNM) to reconstruct the failed sensor data in MIMO radar in order to handle sensor failure in direction-of-arrival (DOA) estimation. First, we formulate a new tensor by convoluting the third-order tensor of MIMO radar with a kernel tensor in a truncated manner and establish a low-rank TC model equipped with the truncated CNNM. Then, we introduce a tractable relaxation of our minimization function by deducing the equality relation between the truncated convolution nuclear norm of a tensor and the nuclear norm of the truncated convolution matrix of the same tensor. Finally, we derive an efficient algorithm implementation based on alternating direction method of multipliers (ADMM) to tackle our problem. Simulation results show that the proposed method is quite effective in completing tensors with structurally missing entries and facilitating more accurate data recovery by exploiting the inherent multidimensional nature. This provides superior DOA estimation performance in MIMO radar.

A Frobenius Norm Regularization Method for Convolutional Kernel Tensors in Neural Networks.

The convolutional neural network is a very important model of deep learning. It can help avoid the exploding/vanishing gradient problem and improve the generalizability of a neural network if the singular values of the Jacobian of a layer are bounded around 1 in the training process. We propose a new Frobenius norm penalty function for a convolutional kernel tensor to let the singular values of the corresponding transformation matrix be bounded around 1. We show how to carry out the gradient-type methods. This provides a potentially useful regularization method for the weights of convolutional layers.

Time-Constrained Adversarial Defense in IoT Edge Devices through Kernel Tensor Decomposition and Multi-DNN Scheduling.

The development of deep learning technology has resulted in great contributions in many artificial intelligence services, but adversarial attack techniques on deep learning models are also becoming more diverse and sophisticated. IoT edge devices take cloud-independent on-device DNN (deep neural network) processing technology to exhibit a fast response time. However, if the computational complexity of the denoizer for adversarial noises is high, or if a single embedded GPU is shared by multiple DNN models, adversarial defense at the on-device level is bound to represent a long latency. To solve this problem, eDenoizer is proposed in this paper. First, it applies Tucker decomposition to reduce the computational amount required for convolutional kernel tensors in the denoizer. Second, eDenoizer effectively orchestrates both the denoizer and the model defended by the denoizer simultaneously. In addition, the priority of the CPU side can be projected onto the GPU which is completely priority-agnostic, so that the delay can be minimized when the denoizer and the defense target model are assigned a high priority. As a result of confirming through extensive experiments, the reduction of classification accuracy was very marginal, up to 1.78%, and the inference speed accompanied by adversarial defense was improved up to 51.72%.

Novel feature selection method via kernel tensor decomposition for improved multi-omics data analysis

BackgroundFeature selection of multi-omics data analysis remains challenging owing to the size of omics datasets, comprising approximately 10^2–10^5 features. In particular, appropriate methods to weight individual omics datasets are unclear, and the approach adopted has substantial consequences for feature selection. In this study, we extended a recently proposed kernel tensor decomposition (KTD)-based unsupervised feature extraction (FE) method to integrate multi-omics datasets obtained from common samples in a weight-free manner.MethodKTD-based unsupervised FE was reformatted as the collection of kernelized tensors sharing common samples, which was applied to synthetic and real datasets.ResultsThe proposed advanced KTD-based unsupervised FE method showed comparative performance to that of the previously proposed KTD method, as well as tensor decomposition-based unsupervised FE, but required reduced memory and central processing unit time. Moreover, this advanced KTD method, specifically designed for multi-omics analysis, attributes P values to features, which is rare for existing multi-omics–oriented methods.ConclusionsThe sample R code is available at https://github.com/tagtag/MultiR/.

Deep neural network compression by Tucker decomposition with nonlinear response

Deep neural networks have shown impressive performance in many areas, including computer vision and natural language processing. Millions of parameters in deep neural network limit its deployment in low-end devices due to intensive memory and expensive computational cost. In the literature, several network compression techniques based on tensor decompositions have been proposed to compress deep neural networks. Existing techniques are designed in each network unit by approximating linear response or kernel tensor using various tensor decomposition methods. What is more, research has shown that there exists significant redundancy between different filters and feature channels of kernel tensor in each convolution layer. In this paper, we propose a new algorithm to compress deep neural network by considering both nonlinear response and the multilinear low-rank constraint in the kernel tensor. To overcome the resulted difficulty of nonconvex optimization, we propose a convex relaxation scheme such that it can be solved by alternating direction method of multipliers (ADMM) directly. Thus, the Tucker-2 rank and the feature matrix of Tucker decomposition can be determined simultaneously. The effectiveness of the proposed method is evaluated on CIFAR-10 and large-scale ILSVRC12 datasets for CNNs including ResNet-18, AlexNet and GoogleNet. According to our numerical computation, the proposed method is able to obtain highly reduction in model size with a small loss in accuracy. The compression performance of the proposed method is better than existing methods.

Verified tensor-program optimization via high-level scheduling rewrites

We present a lightweight Coq framework for optimizing tensor kernels written in a pure, functional array language. Optimizations rely on user scheduling using series of verified, semantics-preserving rewrites. Unusually for compilation targeting imperative code with arrays and nested loops, all rewrites are source-to-source within a purely functional language. Our language comprises a set of core constructs for expressing high-level computation detail and a set of what we call reshape operators, which can be derived from core constructs but trigger low-level decisions about storage patterns and ordering. We demonstrate that not only is this system capable of deriving the optimizations of existing state-of-the-art languages like Halide and generating comparably performant code, it is also able to schedule a family of useful program transformations beyond what is reachable in Halide.

Fractional stochastic partial differential equation for random tangent fields on the sphere

This paper develops a fractional stochastic partial differential equation (SPDE) to model the evolution of a random tangent vector field on the unit sphere. The SPDE is governed by a fractional diffusion operator to model the Levy-type behaviour of the spatial solution, a fractional derivative in time to depict the intermittency of its temporal solution, and is driven by vector-valued fractional Brownian motion on the unit sphere to characterize its temporal long-range dependence. The solution to the SPDE is presented in the form of the Karhunen-Loeve expansion in terms of vector spherical harmonics. Its covariance matrix function is established as a tensor field on the unit sphere that is an expansion of Legendre tensor kernels. The variance of the increments and approximations to the solutions are studied and convergence rates of the approximation errors are given. It is demonstrated how these convergence rates depend on the decay of the power spectrum and variances of the fractional Brownian motion.

Identification of genes associated with altered gene expression and m6A profiles during hypoxia using tensor decomposition based unsupervised feature extraction

Although hypoxia is a critical factor that can drive the progression of various diseases, the mechanism underlying hypoxia itself remains unclear. Recently, m6A has been proposed as an important factor driving hypoxia. Despite successful analyses, potential genes were not selected with statistical significance but were selected based solely on fold changes. Because the number of genes is large while the number of samples is small, it was impossible to select genes using conventional feature selection methods with statistical significance. In this study, we applied the recently proposed principal component analysis (PCA), tensor decomposition (TD), and kernel tensor decomposition (KTD)-based unsupervised feature extraction (FE) to a hypoxia data set. We found that PCA, TD, and KTD-based unsupervised FE could successfully identify a limited number of genes associated with altered gene expression and m6A profiles, as well as the enrichment of hypoxia-related biological terms, with improved statistical significance.

Mathematical formulation and application of kernel tensor decomposition based unsupervised feature extraction

In this work, we extended the recently developed tensor decomposition (TD) based unsupervised feature extraction (FE) to a kernel-based method through a mathematical formulation. Subsequently, the kernel TD (KTD) based unsupervised FE was applied to two synthetic examples as well as real data sets, and the findings were compared with those obtained previously using the TD-based unsupervised FE approaches. The KTD-based unsupervised FE outperformed or performed comparably with the TD-based unsupervised FE in large p small n situations, which are situations involving a limited number of samples with many variables (observations). Nevertheless, the KTD-based unsupervised FE outperformed the TD-based unsupervised FE in non large p small n situations. In general, although the use of the kernel trick can help the TD-based unsupervised FE gain more variations, a wider range of problems may also be encountered. Considering the outperformance or comparable performance of the KTD-based unsupervised FE compared to the TD-based unsupervised FE when applied to large p small n problems, it is expected that the KTD-based unsupervised FE can be applied in the genomic science domain, which involves many large p small n problems, and, in which, the TD-based unsupervised FE approach has been effectively applied.