Sparse Tensor Research Articles

Collocation methods for nonlinear differential equations on low-rank manifolds

We introduce new methods for integrating nonlinear differential equations on low-rank manifolds. These methods rely on interpolatory projections onto the tangent space, enabling low-rank time integration of vector fields that can be evaluated entry-wise. A key advantage of our approach is that it does not require the vector field to exhibit low-rank structure, thereby overcoming significant limitations of traditional dynamical low-rank methods based on orthogonal projection. To construct the interpolatory projectors, we develop a sparse tensor sampling algorithm based on the discrete empirical interpolation method (DEIM) that parameterizes tensor train manifolds and their tangent spaces with cross interpolation. Using these projectors, we propose two time integration schemes on low-rank tensor train manifolds. The first scheme integrates the solution at selected interpolation indices and constructs the solution with cross interpolation. The second scheme generalizes the well-known orthogonal projector-splitting integrator to interpolatory projectors. We demonstrate the proposed methods with applications to several tensor differential equations arising from the discretization of partial differential equations.

STCO: Enhancing Training Efficiency via Structured Sparse Tensor Compilation Optimization

Network sparsification serves as an effective technique to accelerate Deep Neural Network (DNN) inference. However, existing sparsification techniques often rely on structured sparsity, which yields limited benefits. This is primarily due to the significant memory and computational overhead introduced by numerous sparse storage formats during address generation and gradient updates. Additionally, many of these solutions are tailored solely for the inference phase, neglecting the crucial training phase. In this paper, we introduce STCO, a novel Sparse Tensor Compilation Optimization technique that significantly enhances training efficiency through structured sparse tensor compilation. Central to STCO is the Tensorization-aware Index Entity (TIE) format, which effectively represents structured sparse tensors by eliminating redundant indices and minimizing storage overhead. The TIE format plays a pivotal role in the Address-carry flow (AC flow) pass, which optimizes the data layout at the computational graph level. This pass leverages the TIE format to enhance the efficiency of tensor representations, enabling more compact and efficient sparse tensor storage. Meanwhile, a shape inference pass utilizes the AC flow to derive optimized tensor shapes, further refining the performance of sparse tensor operations. Moreover, the Address-Carry TIE Flow dynamically tracks nonzero addresses, extending the benefits of sparse optimization to both forward and backward propagation. This seamless integration into the training pipeline enables a smooth transition to sparse tensor compilation without significant modifications to existing codebases. To further boost training performance, we implement an operator-level AC flow optimization pass tailored for structured sparse tensors. This pass generates efficient addresses, ensuring minimal computational overhead during sparse tensor operations. The flexibility of STCO allows it to be efficiently integrated into various frameworks or compilers, providing a robust solution for enhancing training efficiency with structured sparse tensors. Experiments demonstrated that STCO achieved impressive speedups of 3.64 ×, 5.43 ×, 4.89 ×, and 3.91 × when compared to state-of-the-art sparse formats on VGG16, ResNet-18, MobileNetV1, and MobileNetV2, respectively. These findings underscore the efficiency and superiority of our proposed approach in leveraging unstructured sparsity for Deep Neural Network inference acceleration.

SparseAuto: An Auto-scheduler for Sparse Tensor Computations using Recursive Loop Nest Restructuring

Automated code generation and performance enhancements for sparse tensor algebra have become essential in many real-world applications, such as quantum computing, physical simulations, computational chemistry, and machine learning. General sparse tensor algebra compilers are not always versatile enough to generate asymptotically optimal code for sparse tensor contractions. This paper shows how to generate asymptotically better schedules for complex sparse tensor expressions using kernel fission and fusion. We present generalized loop restructuring transformations to reduce asymptotic time complexity and memory footprint. Furthermore, we present an auto-scheduler that uses a partially ordered set (poset)-based cost model that uses both time and auxiliary memory complexities to prune the search space of schedules. In addition, we highlight the use of Satisfiability Module Theory (SMT) solvers in sparse auto-schedulers to approximate the Pareto frontier of better schedules to the smallest number of possible schedules, with user-defined constraints available at compile-time. Finally, we show that our auto-scheduler can select better-performing schedules and generate code for them. Our results show that the auto-scheduler provided schedules achieve orders-of-magnitude speedup compared to the code generated by the Tensor Algebra Compiler (TACO) for several computations on different real-world tensors.

Compiler Support for Sparse Tensor Convolutions

This paper extends prior work on sparse tensor algebra compilers to generate asymptotically efficient code for tensor expressions with affine subscript expressions. Our technique enables compiler support for a wide range of sparse computations, including sparse convolutions and pooling that are widely used in ML and graphics applications. We propose an approach that gradually rewrites compound subscript expressions to simple subscript expressions with loops that exploit the sparsity pattern of the input sparse tensors. As a result, the time complexity of the generated kernels is bounded by the number of stored elements and not by the shape of the tensors. Our approach seamlessly integrates into existing frameworks and is compatible with recent advances in compilers for sparse computations, including the flexibility to efficiently handle arbitrary combinations of different sparse tensor formats. The implementation of our algorithm is open source and upstreamed to the MLIR sparse compiler. Experimental results show that our method achieves 19.5x speedup when compared with the state-of-the-art compiler-based method at 99.9% sparsity. The generated sparse kernels start to outperform dense convolution implementations at about 80% sparsity.

FuBay: An Integrated Fusion Framework for Hyperspectral Super-Resolution Based on Bayesian Tensor Ring.

Fusion with corresponding finer-resolution images has been a promising way to enhance hyperspectral images (HSIs) spatially. Recently, low-rank tensor-based methods have shown advantages compared with other kind of ones. However, these current methods either relent to blind manual selection of latent tensor rank, whereas the prior knowledge about tensor rank is surprisingly limited, or resort to regularization to make the role of low rankness without exploration on the underlying low-dimensional factors, both of which are leaving the computational burden of parameter tuning. To address that, a novel Bayesian sparse learning-based tensor ring (TR) fusion model is proposed, named as FuBay. Through specifying hierarchical sprasity-inducing prior distribution, the proposed method becomes the first fully Bayesian probabilistic tensor framework for hyperspectral fusion. With the relationship between component sparseness and the corresponding hyperprior parameter being well studied, a component pruning part is established to asymptotically approaching true latent rank. Furthermore, a variational inference (VI)-based algorithm is derived to learn the posterior of TR factors, circumventing nonconvex optimization that bothers the most tensor decomposition-based fusion methods. As a Bayesian learning methods, our model is characterized to be parameter tuning-free. Finally, extensive experiments demonstrate its superior performance when compared with state-of-the-art methods.

Low-rank sparse fully-connected tensor network for tensor completion

Fully-connected tensor network (FCTN) has recently drawn lots of attention in tensor completion due to its full description of all correlations between any two modes. However, the FCTN model has multiple ranks, and existing methods often ignore the difficulty brought about by rank selection, especially when the model is rank-sensitive. To overcome this drawback, this paper proposed a low-rank sparse FCTN for tensor completion. Specifically, we theoretically show that, for a tensor with FCTN structure, its subtensor can be represented as a coefficient sum of basic tensors, where the coefficients are remained in the FCTN’s factors. This means that factors’ sparsity can improve the robustness of FCTN to larger rank selection. Moreover, according to the relation between the target tensor and its FCTN’s factors, low-rank constrain is used to enhance rank robustness. Lastly, we optimize the proposed model by the alternating direction method of multipliers (ADMM) algorithm. Experimental results show that the low-rank sparse constraint effectively improves the rank robustness of the FCTN model, and achieves excellent results compared with other state-of-the-art completion methods.

Deep learning for symmetry classification using sparse 3D electron density data for inorganic compounds

We report a novel deep learning (DL) method for classifying inorganic compounds using 3D electron density data. We transform Density Functional Theory (DFT)-derived CHGCAR files from the Materials Project (MP) and experimental data from the Inorganic Crystal Structure Database (ICSD) into point clouds and sparse tensors, optimized for use in DL models such as PointNet and Sparse 3D CNN. This approach effectively overcomes the limitations of handling the dense 3D data, a common challenge in DL. Contrasting with traditional 1D or 2D X-ray diffraction (XRD) patterns that necessitate complex reciprocal space analysis, our method utilizes 3D density data for direct interpretation in real lattice space. This shift significantly enhances classification accuracy, outperforming traditional XRD-driven DL methods. We achieve accuracies of 97.28%, 90.77%, and 90.10% for crystal system, extinction group, and space group classifications, respectively. Our 3D electron density-based DL approach not only showcases improved accuracy but also contributes a more intuitive and effective framework for materials discovery.

D-nontrilocality of sparse probability tensors and the triangle network

Abstract Bell’s inequalities are linear and apply for cases of two entangled bodies. In this work, we consider the case of entanglement among three bodies as previously discussed in [Renou, et al Phys. Rev. Lett., 123, 140 401 (2019)] and based on triangle network. By discussing the question whether a sparse probability tensor (SPT) can be represented by a discrete trilocal hidden variable model (D-triLHVM), we show that every SPT having a D-triLHVM satisfies a set of concrete equalities and a nonlinear inequality, which can be used to detect whether a D-triLHVM can describe the network completely. As an application, we re-explore the D-nontrilocality of the correlations studied by Renou et al and that of the triangle network with shared entangled pure states. We also leave open questions about the closednees of the set of all D-trilocal probability tensors and the description with a continuous trilocal hidden variable model.

Sparse Tensor Decomposition of Multi-Sensory Data for Fault Localization in Rotating Machinery Health Monitoring

Multi-sensor monitoring is prevalent in modern structural health monitoring (SHM) practice. As the number of sensors and sampling requirements increase, a monitoring sensor network can generate substantial data which are high-volume and high-dimensional, especially for large structure and machinery. In condition-based monitoring (CBM) of rotating machines, e.g., gas turbine, rotor fault diagnosis serves a significant role in the system reliability, safety, and efficiency, which helps reduce potential damage to the on-rotor structures and avoid catastrophic failures. For multi-channel vibration measurement, classical rotor diagnosis approaches typically involve data fusion techniques based on cross spectral analysis for identifying spectral correlation, e.g., cross power spectral density, and/or matrix analysis tool for dimensionality reduction, e.g., principal component analysis. The operation on vectors or matrices may limit their effectiveness for higher-order array or tensor data. To circumvent this limitation, the third order vibrational spectral tensor is generated by representing the multi-channel acceleration spectra as the three-way array. Through nonnegative Tucker decomposition (NTD), the spectral tensor is decomposed into multiple principal factors: the spectral factor, segmental factor, channel-wise factor, and the dense tensor core. The correlations across the characteristic spectral contents and the sensor channels are revealed by the factorization, which enables the diagnosis of rotor fault and facilitates fault localization by identifying the dominant channels of the characteristic spectral factor. The method is validated on an experimental rotor testbed where the faulty channel with crack, rub, or misalignment fault, is effectively localized via 4-channel vibration measurements, which presents a promising approach for multi-sensor fusion and fault diagnosis in rotating machinery health monitoring.

Snow-CLOCs: Camera-LiDAR Object Candidate Fusion for 3D Object Detection in Snowy Conditions.

Although existing 3D object-detection methods have achieved promising results on conventional datasets, it is still challenging to detect objects in data collected under adverse weather conditions. Data distortion from LiDAR and cameras in such conditions leads to poor performance of traditional single-sensor detection methods. Multi-modal data-fusion methods struggle with data distortion and low alignment accuracy, making accurate target detection difficult. To address this, we propose a multi-modal object-detection algorithm, Snow-CLOCs, specifically for snowy conditions. In image detection, we improved the YOLOv5 algorithm by integrating the InceptionNeXt network to enhance feature extraction and using the Wise-IoU algorithm to reduce dependency on high-quality data. For LiDAR point-cloud detection, we built upon the SECOND algorithm and employed the DROR filter to remove noise, enhancing detection accuracy. We combined the detection results from the camera and LiDAR into a unified detection set, represented using a sparse tensor, and extracted features through a 2D convolutional neural network to achieve object detection and localization. Snow-CLOCs achieved a detection accuracy of 86.61% for vehicle detection in snowy conditions.

Efficient Utilization of Multi-Threading Parallelism on Heterogeneous Systems for Sparse Tensor Contraction

Efficient Utilization of Multi-Threading Parallelism on Heterogeneous Systems for Sparse Tensor Contraction

Simple proofs for lattice coverings and sparse tensors

We provide simple proofs of analogues for coverings numbers of lattices of several recently studied basic statements on the ranks of tensors. These analogues involve in particular the interplay between different covering numbers, the existence of sublattices with high covering number and satisfying various conditions, and upper bounds on the size of the set of possible coverings with minimal size. We highlight the differences and analogies between the proofs in both settings.

CuFastTucker: A Novel Sparse FastTucker Decomposition For HHLST on Multi-GPUs

High-order, high-dimension, and large-scale sparse tensors (HHLST) have found their origin in various real industrial applications, such as social networks, recommender systems, bioinformatics, and traffic information. To handle these complex tensors, sparse tensor decomposition techniques are employed to project the HHLST into a low-rank space. In this paper, we propose a novel sparse tensor decomposition model called Sparse FastTucker Decomposition (SFTD), which is a variant of Sparse Tucker Decomposition (STD). The SFTD utilizes Kruskal approximation for the core tensor, and we present a theorem that reduces the exponential space and computational overhead to a polynomial one. Additionally, we reduce the space overhead of intermediate parameters in the algorithmic process by sampling the intermediate matrix. Furthermore, this method guarantees convergence. To enhance the speed of SFTD, we leverage the compactness of matrix multiplication and parallel access through a stochastic strategy, resulting in GPU-accelerated cuFastTucker. Moreover, we propose a data division and communication strategy for cuFastTucker to accommodate data on Multi-GPU setups. Our proposed cuFastTucker demonstrates faster calculation and convergence speeds, as well as significantly lower space and computational overhead compared to state-of-the-art (SOTA) algorithms such as P-Tucker, Vest, GTA, Bigtensor and SGD_Tucker.

UniSparse: An Intermediate Language for General Sparse Format Customization

The ongoing trend of hardware specialization has led to a growing use of custom data formats when processing sparse workloads, which are typically memory-bound. These formats facilitate optimized software/hardware implementations by utilizing sparsity pattern- or target-aware data structures and layouts to enhance memory access latency and bandwidth utilization. However, existing sparse tensor programming models and compilers offer little or no support for productively customizing the sparse formats. Additionally, because these frameworks represent formats using a limited set of per-dimension attributes, they lack the flexibility to accommodate numerous new variations of custom sparse data structures and layouts. To overcome this deficiency, we propose UniSparse, an intermediate language that provides a unified abstraction for representing and customizing sparse formats. Unlike the existing attribute-based frameworks, UniSparse decouples the logical representation of the sparse tensor (i.e., the data structure) from its low-level memory layout, enabling the customization of both. As a result, a rich set of format customizations can be succinctly expressed in a small set of well-defined query, mutation, and layout primitives. We also develop a compiler leveraging the MLIR infrastructure, which supports adaptive customization of formats, and automatic code generation of format conversion and compute operations for heterogeneous architectures. We demonstrate the efficacy of our approach through experiments running commonly-used sparse linear algebra operations with specialized formats on multiple different hardware targets, including an Intel CPU, an NVIDIA GPU, an AMD Xilinx FPGA, and a simulated processing-in-memory (PIM) device.

Compiling Recurrences over Dense and Sparse Arrays

We present a framework for compiling recurrence equations into native code. In our framework, users specify a system of recurrences, the types of data structures that store inputs and outputs, and scheduling commands for optimization. Our compiler then lowers these specifications into native code that respects the dependencies in the recurrence equations. Our compiler can generate code over both sparse and dense data structures, and determines if the recurrence system is solvable with the provided scheduling primitives. We evaluate the performance and correctness of the generated code on several recurrences, from domains as diverse as dense and sparse matrix solvers, dynamic programming, graph problems, and sparse tensor algebra. We demonstrate that the generated code has competitive performance to hand-optimized implementations in libraries. However, these handwritten libraries target specific recurrences, specific data structures, and specific optimizations. Our system, on the other hand, automatically generates implementations from recurrences, data formats, and schedules, giving our system more generality than library approaches.

Bayesian fusion of multispectral and panchromatic images using a multi-mode and multiorder gradient tensor-based l 1 / 2 sparse model

ABSTRACT In this letter, based on the tensor representation modelling, we propose a multi-mode and multi-order gradient tensor-based non-convex model (M2GTNM) for Bayesian fusion of multispectral (MS) and panchromatic (Pan) images, which aims at producing the high-resolution MS (HRMS) images. Specifically, by modelling the MS image as the order-3 tensor, we mainly develop the multi-mode and multi-order gradient tensor sparse priors of MS image for fusion. For the spectral preservation of low-resolution MS (LRMS) image, the spectral fidelity constraint between HRMS and LRMS images is imposed. For the spatial-mode prior modelling, the multi-order spatial gradient tensor-based non-convex l 1 / 2 sparse prior between HRMS and Pan images is particularly imposed. Moreover, for the spectral-mode prior modelling, the spectral gradient tensor-based non-convex l 1 / 2 sparse prior between HRMS and upsampled LRMS images is further imposed. Then, we apply the alternating direction method of multipliers to optimize the proposed model. Finally, the reduced-scale and full-scale fusion experiments both validate the effectiveness of M2GTNM method.

Saliency-Guided Sparse Low-Rank Tensor Approximation for Unsupervised Anomaly Detection of Hyperspectral Remote Sensing Images

Hyperspectral anomaly detection can separate sparse anomalies from the low-rank background component under an unsupervised behavior due to sufficient spectral information. Therefore, hyperspectral image anomaly detection technology has great application potential and value in public security and national defense. Currently, most existing models attempt to detect anomalous targets with a sparsity prior, without further considering the visual saliency of the targets themselves. To tackle this issue, this paper proposes a saliency-guided sparse low-rank tensor approximation model, called SSLR, to detect anomalous targets from hyperspectral remote sensing images in an unsupervised manner. Specifically, we first explore the saliency information of each pixel for regularizing the sparse anomaly matrix. We then suggest a three-directional tensor nuclear norm to obtain a low-rank background to characterize the background component. We solve the SSLR optimization problem by an efficient alternating direction method of multipliers framework. Experiments conducted on benchmark hyperspectral datasets demonstrate that the proposed SSLR outperforms some state-of-the-art anomaly detection methods.

Optimal evaluation of symmetry-adapted n-correlations via recursive contraction of sparse symmetric tensors

Abstract We present a comprehensive analysis of an algorithm for evaluating high-dimensional polynomials that are invariant (or equi-variant) under permutations and rotations. This task arises in the evaluation linear models as well as equivariant neural network models of many-particle systems. The theoretical bottleneck is the contraction of a high-dimensional symmetric and sparse tensor with a specific sparsity pattern that is directly related to the symmetries imposed on the polynomial. The sparsity of this tensor makes it challenging to construct a highly efficient evaluation scheme. The references [10, 11] introduced a recursive evaluation strategy that relied on a number of heuristics, but performed well in tests. In the present work, we propose an explicit construction of such a recursive evaluation strategy and show that it is in fact optimal in the limit of infinite polynomial degree.

Harnessing Manycore Processors with Distributed Memory for Accelerated Training of Sparse and Recurrent Models

Current AI training infrastructure is dominated by single instruction multiple data (SIMD) and systolic array architectures, such as Graphics Processing Units (GPUs) and Tensor Processing Units (TPUs), that excel at accelerating parallel workloads and dense vector matrix multiplications. Potentially more efficient neural network models utilizing sparsity and recurrence cannot leverage the full power of SIMD processor and are thus at a severe disadvantage compared to today's prominent parallel architectures like Transformers and CNNs, thereby hindering the path towards more sustainable AI. To overcome this limitation, we explore sparse and recurrent model training on a massively parallel multiple instruction multiple data (MIMD) architecture with distributed local memory. We implement a training routine based on backpropagation though time (BPTT) for the brain-inspired class of Spiking Neural Networks (SNNs) that feature binary sparse activations. We observe a massive advantage in using sparse activation tensors with a MIMD processor, the Intelligence Processing Unit (IPU) compared to GPUs. On training workloads, our results demonstrate 5-10x throughput gains compared to A100 GPUs and up to 38x gains for higher levels of activation sparsity, without a significant slowdown in training convergence or reduction in final model performance. Furthermore, our results show highly promising trends for both single and multi IPU configurations as we scale up to larger model sizes. Our work paves the way towards more efficient, non-standard models via AI training hardware beyond GPUs, and competitive large scale SNN models.

STile: Searching Hybrid Sparse Formats for Sparse Deep Learning Operators Automatically

Sparse operators, i.e., operators that take sparse tensors as input, are of great importance in deep learning models. Due to the diverse sparsity patterns in different sparse tensors, it is challenging to optimize sparse operators by seeking an optimal sparse format, i.e., leading to the lowest operator latency. Existing works propose to decompose a sparse tensor into several parts and search for a hybrid of sparse formats to handle diverse sparse patterns. However, they often make a trade-off between search space and search time: their search spaces are limited in some cases, resulting in limited operator running efficiency they can achieve. In this paper, we try to extend the search space in its breadth (by doing flexible sparse tensor transformations) and depth (by enabling multi-level decomposition). We formally define the multi-level sparse format decomposition problem, which is NP-hard, and we propose a framework STile for it. To search efficiently, a greedy algorithm is used, which is guided by a cost model about the latency of computing a sub-task of the original operator after decomposing the sparse tensor. Experiments of two common kinds of sparse operators, SpMM and SDDMM, are conducted on various sparsity patterns, and we achieve 2.1-18.0× speedup against cuSPARSE on SpMMs and 1.5 - 6.9× speedup against DGL on SDDMM. The search time is less than one hour for any tested sparse operator, which can be amortized.