Sparse Representation Of Data Research Articles

Cerberus: Triple Mode Acceleration of Sparse Matrix and Vector Multiplication

The multiplication of sparse matrix and vector (SpMV) is one of the most widely used kernels in high-performance computing as well as machine learning acceleration for sparse neural networks. The design space of SpMV accelerators has two axes: algorithm and matrix representation. There have been two widely used algorithms and data representations. Two algorithms, scalar multiplication and dot product, can be combined with two sparse data representations, compressed sparse and bitmap formats for the matrix and vector. Although the prior accelerators adopted one of the possible designs, it is yet to be investigated which design is the best one across different hardware resources and workload characteristics. This paper first investigates the impact of design choices with respect to the algorithm and data representation. Our evaluation shows that no single design always outperforms the others across different workloads, but the two best designs (i.e. compressed sparse format and bitmap format with dot product) have complementary performance with trade-offs incurred by the matrix characteristics. Based on the analysis, this study proposes Cerberus, a triple-mode accelerator supporting two sparse operation modes in addition to the base dense mode. To allow such multi-mode operation, it proposes a prediction model based on matrix characteristics under a given hardware configuration, which statically selects the best mode for a given sparse matrix with its dimension and density information. Our experimental results show that Cerberus provides 12.1 × performance improvements from a dense-only accelerator, and 1.5 × improvements from a fixed best SpMV design.

Method for Sparse Representation of Complex Data Based on Overcomplete Basis, l1 Norm, and Neural MFNN-like Network

The article presents the results of research into a method for representing complex data based on an overcomplete basis and l0/l1 norms. The proposed method is an extended modification of the neural-like MFNN (minimum fuel neural network) for the case of complex data. The influence of the choice of activation function on the performance of the method is analyzed. The results of the numerical simulation demonstrate the effectiveness of the proposed method for the case of sparse representation of complex data and can be used to determine the direction of arrival (DOA) for a uniform linear array (ULA).

Sparse Regularization Based on Orthogonal Tensor Dictionary Learning for Inverse Problems

In seismic data processing, data recovery including reconstruction of the missing trace and removal of noise from the recorded data are the key steps in improving the signal-to-noise ratio (SNR). The reconstruction of seismic data and removal of noise becomes a sparse optimization problem that can be solved by using sparse regularization. Sparse regularization is a key tool in the solution of inverse problems. They are used to introduce prior knowledge and make the approximation of ill-posed inverses feasible. It deals with ill-posedness by replacing an ill-posed inverse problem with a well-posed problem that has a solution close to the true solution. In the last 2 decades, interest has shifted from linear toward nonlinear regularization methods even for linear inverse problems. In inverse problems, regularizations serve as stabilizing the solution of ill-posed inverse problems and give a solution that adequately fits measurements without producing unjustifiably complex artifacts. In this paper, we present a novel sparse regularization based on a tensor-based dictionary method for inverse problems (seismic data interpolation and denoising). This regularization avoids the vectorization step for sparse representation of seismic data during the reconstruction process. The key step in sparsifying signals is the choice of sparsity-promoting dictionary learning. The learning-based approach can adaptively sparsify datasets but has high computational complexity and involves no prior-constraint pattern information for the dataset. Many existing dictionary learning methods would be computationally infeasible for the high dimensional seismic data processing. These methods also destroy the essential information as well as it reduces the discriminability and expressibility of the signal, since they deal with vectorization. In this paper, the orthogonal tensor dictionary learning that learns a dictionary from the input data by employing orthogonality and separability is proposed as sparse regularization for the inverse problems. The performance of the proposed method was validated in seismic data interpolation and denoising individually as well as simultaneously. Numerical examples of synthetic and real seismic datasets demonstrate the validity of the proposed method. The SNR of the recovered data confirms that the proposed method is the most effective method than K-singular value decomposition and orthogonal dictionary learning methods.

Self-adaptive seismic data reconstruction and denoising using dictionary learning based on morphological component analysis

Data reconstruction and data denoising are two critical preliminary steps in seismic data processing. Compressed Sensing states that a signal can be recovered by a series of solving algorithms if it is sparse in a transform domain, and has been well applied in the field of reconstruction, when, sparse representation of seismic data is the key point. Considering the complexity and diversity of seismic data, a single mathematical transformation will lead to incomplete sparse expression and bad restoration effects. Morphological Component Analysis (MCA) decomposes a signal into several components with outstanding morphological features to approximate the complex internal data structure. However, the representation ability of combined dictionaries is constrained by the number of dictionaries, and cannot be self-adaptively matched with the data features. Dictionary learning overcomes the limitation of fixed base function by training dictionaries that are fully suitable for processed data, but requires huge amount of time and considerable hardware cost. To solve the above problems, a new dictionary library (K-Singluar Value Decomposition learning dictionary and Discrete Cosine Transform dictionary) is hereby proposed based on the efficiency of fixed base dictionary and the high precision of learning dictionary. The self-adaptive sparse representation is achieved under the Morphological Component Analysis framework and is successfully applied to the reconstruction and denoising of seismic data. Real data tests have proved that the proposed method performs better than single mathematical transformation and other combined dictionaries.

Three-dimensional Imaging Method of Target Based on Time-domain Sparse Representation of Multi-view SAR Data

Three-dimensional Imaging Method of Target Based on Time-domain Sparse Representation of Multi-view SAR Data

Connections between Deep Equilibrium and Sparse Representation Models with Application to Hyperspectral Image Denoising.

In this study, the problem of computing a sparse representation of multi-dimensional visual data is considered. In general, such data e.g., hyperspectral images, color images or video data consists of signals that exhibit strong local dependencies. A new computationally efficient sparse coding optimization problem is derived by employing regularization terms that are adapted to the properties of the signals of interest. Exploiting the merits of the learnable regularization techniques, a neural network is employed to act as structure prior and reveal the underlying signal dependencies. To solve the optimization problem Deep unrolling and Deep equilibrium based algorithms are developed, forming highly interpretable and concise deep-learning-based architectures, that process the input dataset in a block-by-block fashion. Extensive simulation results, in the context of hyperspectral image denoising, are provided, which demonstrate that the proposed algorithms outperform significantly other sparse coding approaches and exhibit superior performance against recent state-of-the-art deep-learning-based denoising models. In a wider perspective, our work provides a unique bridge between a classic approach, that is the sparse representation theory, and modern representation tools that are based on deep learning modeling.

Multiple Graph Adaptive Regularized Semi-Supervised Nonnegative Matrix Factorization with Sparse Constraint for Data Representation

Multiple graph and semi-supervision techniques have been successfully introduced into the nonnegative matrix factorization (NMF) model for taking full advantage of the manifold structure and priori information of data to capture excellent low-dimensional data representation. However, the existing methods do not consider the sparse constraint, which can enhance the local learning ability and improve the performance in practical applications. To overcome this limitation, a novel NMF-based data representation method, namely, the multiple graph adaptive regularized semi-supervised nonnegative matrix factorization with sparse constraint (MSNMFSC) is developed in this paper for obtaining the sparse and discriminative data representation and increasing the quality of decomposition of NMF. Particularly, based on the standard NMF, the proposed MSNMFSC method combines the multiple graph adaptive regularization, the limited supervised information and the sparse constraint together to learn the more discriminative parts-based data representation. Moreover, the convergence analysis of MSNMFSC is studied. Experiments are conducted on several practical image datasets in clustering tasks, and the clustering results have shown that MSNMFSC achieves better performance than several most related NMF-based methods.

Seismic data denoising using curvelet transforms and fast non-local means

Spectral transform, known as a curvelet, enables sparse representations of complex data. Denoising wave propagation in disordered media and pattern recognition are just a few of the numerous domains in which denoising has a potential application. Based on directional basis functions, this spectral method represents objects with discontinuities along a smooth curve. In this study, we used this technique to eliminate ground roll, an unwanted feature signal that can be seen in seismic data obtained by sonating the earth’s geological formations. Additionally, we improved the curvelet transform threshold denoising method combined with a fast non-local mean to remove seismic random noise. First, cyclic translation and block complex domain threshold methods were introduced into the curvelet transform threshold denoising, and the traditional curvelet threshold denoising method was improved to obtain the best denoising results; Subsequently, the removed noise was filtered using a fast non-local mean method to obtain a valid signal. Finally, the data obtained in the above two steps were added to obtain the final denoising result. The results of the model tests and actual seismic data denoising showed that the denoising results obtained using this method had a higher signal-to-noise ratio and fidelity than that using the other methods.

Spike-Timing-Dependent Plasticity With Activation-Dependent Scaling for Receptive Fields Development.

Spike-timing-dependent plasticity (STDP) is one of the most popular and deeply biologically motivated forms of unsupervised Hebbian-type learning. In this article, we propose a variant of STDP extended by an additional activation-dependent scale factor. The consequent learning rule is an efficient algorithm, which is simple to implement and applicable to spiking neural networks (SNNs). It is demonstrated that the proposed plasticity mechanism combined with competitive learning can serve as an effective mechanism for the unsupervised development of receptive fields (RFs). Furthermore, the relationship between synaptic scaling and lateral inhibition is explored in the context of the successful development of RFs. Specifically, we demonstrate that maintaining a high level of synaptic scaling followed by its rapid increase is crucial for the development of neuronal mechanisms of selectivity. The strength of the proposed solution is assessed in classification tasks performed on the Modified National Institute of Standards and Technology (MNIST) data set with an accuracy level of 94.65% (a single network) and 95.17% (a network committee)-comparable to the state-of-the-art results of single-layer SNN architectures trained in an unsupervised manner. Furthermore, the training process leads to sparse data representation and the developed RFs have the potential to serve as local feature detectors in multilayered spiking networks. We also prove theoretically that when applied to linear Poisson neurons, our rule conserves total synaptic strength, guaranteeing the convergence of the learning process.

Joint History Matching of Multiple Types of Field Data in a 3D Field-Scale Case Study

This work presents an ensemble-based workflow to simultaneously assimilate multiple types of field data in a proper and consistent manner. The aim of using multiple field datasets is to improve the reliability of estimated reservoir models and avoid the underestimation of uncertainties. The proposed framework is based on an integrated history matching workflow, in which reservoir models are conditioned simultaneously on production, tracer and 4D seismic data with the help of three advanced techniques: adaptive localization (for better uncertainty quantification), weight adjustment (for balancing the influence of different types of field data), and sparse data representation (for handling big datasets). The integrated workflow is successfully implemented and tested in a 3D benchmark case with a set of comparison studies (with and without tracer data). The findings of this study indicate that joint history matching using production, tracer and 4D seismic data results in better estimated reservoir models and improved forecast performance. Moreover, the integrated workflow is flexible, and can be extended to incorporate more types of field data for further performance improvement. As such, the findings of this study can help to achieve a better understanding of the impacts of multiple datasets on history matching performance, and the proposed integrated workflow could serve as a useful tool for real field case studies in general.

Revisiting Sequential Information Bottleneck: New Implementation and Evaluation.

We introduce a modern, optimized, and publicly available implementation of the sequential Information Bottleneck clustering algorithm, which strikes a highly competitive balance between clustering quality and speed. We describe a set of optimizations that make the algorithm computation more efficient, particularly for the common case of sparse data representation. The results are substantiated by an extensive evaluation that compares the algorithm to commonly used alternatives, focusing on the practically important use case of text clustering. The evaluation covers a range of publicly available benchmark datasets and a set of clustering setups employing modern word and sentence embeddings obtained by state-of-the-art neural models. The results show that in spite of using the more basic Term-Frequency representation, the proposed implementation provides a highly attractive trade-off between quality and speed that outperforms the alternatives considered. This new release facilitates the use of the algorithm in real-world applications of text clustering.

THETA: A High-Efficiency Training Accelerator for DNNs With Triple-Side Sparsity Exploration

Training deep neural networks (DNNs) on edge devices has attracted increasing attention in real-world applications for domain adaption and privacy protection. However, deploying DNN training on resource-limited edge devices is challenging as there are massive computations and data transportation in training. To address this issue, we propose an energy-efficient training accelerator in this work by employing a hybrid compression strategy. Here, various data redundancies are fully exploited, and the real triple-side sparsity is achieved. Hence, the computational complexity is drastically reduced with negligible accuracy loss across a range of transfer learning tasks. To facilitate triple-side zero-skipping operations during different training stages, we first present a novel sparse data representation and a triple-sparsity index matching scheme. Second, a sparse tensor processing unit (STPU) arranged in a hierarchical structure is developed, which enables a flexible dataflow to process convolutional (Conv) and fully connected (FC) layers with diverse computational patterns throughout the entire training. Third, an auxiliary processing unit (APU) is designed to execute some postprocessing operations, such as rectified linear unit (ReLU) and on-the-fly pruning. Finally, the training accelerator is implemented under Taiwan Semiconductor Manufacturing Company (TSMC) 28-nm process and evaluated on multiple benchmarks. The experimental results show that THETA achieves 7.28–22.32 tera operations per second (TOPS) and 45.24–133.70 TOPS/W in performance and energy efficiency, reducing 40– <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$72\times $ </tex-math></inline-formula> training time and 19– <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$63\times $ </tex-math></inline-formula> energy consumption over dense training, respectively. Compared with the prior art, our design offers <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$1.6\times $ </tex-math></inline-formula> throughput and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$1.9\times $ </tex-math></inline-formula> energy efficiency, respectively.

Volumetric Rendering on Wavelet-Based Adaptive Grid

Numerical modeling of physical phenomena frequently involves processes across a wide range of spatial and temporal scales. In the last two decades, the advancements in wavelet-based numerical methodologies to solve partial differential equations, combined with the unique properties of wavelet analysis to resolve localized structures of the solution on dynamically adaptive computational meshes, make it feasible to perform large-scale numerical simulations of a variety of physical systems on a dynamically adaptive computational mesh that changes both in space and time. Volumetric visualization of the solution is an essential part of scientific computing, yet the existing volumetric visualization techniques do not take full advantage of multi-resolution wavelet analysis and are not fully tailored for visualization of a compressed solution on the wavelet-based adaptive computational mesh. Our objective is to explore the alternatives for the visualization of time-dependent data on space-time varying adaptive mesh using volume rendering while capitalizing on the available sparse data representation. Two alternative formulations are explored. The first one is based on volumetric ray casting of multi-scale datasets in wavelet space. Rather than working with the wavelets at the finest possible resolution, a partial inverse wavelet transform is performed as a preprocessing step to obtain scaling functions on a uniform grid at a user-prescribed resolution. As a result, a solution in physical space is represented by a superposition of scaling functions on a coarse regular grid and wavelets on an adaptive mesh. An efficient and accurate ray casting algorithm is based just on these coarse scaling functions. Additional details are added during the ray tracing by taking an appropriate number of wavelets into account based on support overlap with the interpolation point, wavelet coefficient magnitude, and other characteristics, such as opacity accumulation (front to back ordering) and deviation from frontal viewing direction. The second approach is based on complementing of wavelet-based adaptive mesh to the traditional Adaptive Mesh Refinement (AMR) mesh. Both algorithms are illustrated and compared to the existing volume visualization software for Rayleigh-Benard thermal convection and electron density data sets in terms of rendering time and visual quality for different data compression of both wavelet-based and AMR adaptive meshes.

Compressed sensing based dynamic MR image reconstruction by using 3D-total generalized variation and tensor decomposition: k-t TGV-TD

PurposeCompressed Sensing Magnetic Resonance Imaging (CS-MRI) is a promising technique to accelerate dynamic cardiac MR imaging (DCMRI). For DCMRI, the CS-MRI usually exploits image signal sparsity and low-rank property to reconstruct dynamic images from the undersampled k-space data. In this paper, a novel CS algorithm is investigated to improve dynamic cardiac MR image reconstruction quality under the condition of minimizing the k-space recording.MethodsThe sparse representation of 3D cardiac magnetic resonance data is implemented by synergistically integrating 3D total generalized variation (3D-TGV) algorithm and high order singular value decomposition (HOSVD) based Tensor Decomposition, termed k-t TGV-TD method. In the proposed method, the low rank structure of the 3D dynamic cardiac MR data is performed with the HOSVD method, and the localized image sparsity is achieved by the 3D-TGV method. Moreover, the Fast Composite Splitting Algorithm (FCSA) method, combining the variable splitting with operator splitting techniques, is employed to solve the low-rank and sparse problem. Two different cardiac MR datasets (cardiac perfusion and cine MR datasets) are used to evaluate the performance of the proposed method.ResultsCompared with the state-of-art methods, such as k-t SLR, 3D-TGV, HOSVD based tensor decomposition and low-rank plus sparse method, the proposed k-t TGV-TD method can offer improved reconstruction accuracy in terms of higher peak SNR (PSNR) and structural similarity index (SSIM). The proposed k-t TGV-TD method can achieve significantly better and stable reconstruction results than state-of-the-art methods in terms of both PSNR and SSIM, especially for cardiac perfusion MR dataset.ConclusionsThis work proved that the k-t TGV-TD method was an effective sparse representation way for DCMRI, which was capable of significantly improving the reconstruction accuracy with different acceleration factors.

Sparse representations of high dimensional neural data

Conventional Vector Autoregressive (VAR) modelling methods applied to high dimensional neural time series data result in noisy solutions that are dense or have a large number of spurious coefficients. This reduces the speed and accuracy of auxiliary computations downstream and inflates the time required to compute functional connectivity networks by a factor that is at least inversely proportional to the true network density. As these noisy solutions have distorted coefficients, thresholding them as per some criterion, statistical or otherwise, does not alleviate the problem. Thus obtaining a sparse representation of such data is important since it provides an efficient representation of the data and facilitates its further analysis. We propose a fast Sparse Vector Autoregressive Greedy Search (SVARGS) method that works well for high dimensional data, even when the number of time points is relatively low, by incorporating only statistically significant coefficients. In numerical experiments, our methods show high accuracy in recovering the true sparse model. The relative absence of spurious coefficients permits accurate, stable and fast evaluation of derived quantities such as power spectrum, coherence and Granger causality. Consequently, sparse functional connectivity networks can be computed, in a reasonable time, from data comprising tens of thousands of channels/voxels. This enables a much higher resolution analysis of functional connectivity patterns and community structures in such large networks than is possible using existing time series methods. We apply our method to EEG data where computed network measures and community structures are used to distinguish emotional states as well as to ADHD fMRI data where it is used to distinguish children with ADHD from typically developing children.

Co-sparse Non-negative Matrix Factorization.

Non-negative matrix factorization, which decomposes the input non-negative matrix into product of two non-negative matrices, has been widely used in the neuroimaging field due to its flexible interpretability with non-negativity property. Nowadays, especially in the neuroimaging field, it is common to have at least thousands of voxels while the sample size is only hundreds. The non-negative matrix factorization encounters both computational and theoretical challenge with such high-dimensional data, i.e., there is no guarantee for a sparse and part-based representation of data. To this end, we introduce a co-sparse non-negative matrix factorization method to high-dimensional data by simultaneously imposing sparsity in both two decomposed matrices. Instead of adding some sparsity induced penalty such as l1 norm, the proposed method directly controls the number of non-zero elements, which can avoid the bias issues and thus yield more accurate results. We developed an alternative primal-dual active set algorithm to derive the co-sparse estimator in a computationally efficient way. The simulation studies showed that our method achieved better performance than the state-of-art methods in detecting the basis matrix and recovering signals, especially under the high-dimensional scenario. In empirical experiments with two neuroimaging data, the proposed method successfully detected difference between Alzheimer's patients and normal person in several brain regions, which suggests that our method may be a valuable toolbox for neuroimaging studies.

Leak Identification Based on CS-ResNet Under Different Leakage Apertures for Water-Supply Pipeline

Considering the problem of difficulty in transmission and storage due to a large amount of data in the water-supply network monitoring system based on a wireless sensor network (WSN), we propose a sparse representation of the water-supply network monitoring data by using compressed sensing (CS) method. At the same time, aiming at the problem of low leakage identification accuracy caused by information loss under compressed sensing, we propose a leak identification method for a water-supply pipe network based on compressed sensing and deep residual neural network (ResNet). Firstly, under the condition that the observation matrix ensures the integrity of signal information, the compressed sensing theory is used to compress and observe leakage signals to obtain observation data, to reduce the redundant information and volume of the data. At the same time, the observation data is preprocessed to realize the transformation of a one-dimensional signal to a two-dimensional matrix. Then the residual neural network is trained by using the two-dimensional data to realize the automatic, efficient, and accurate leak identification under different leakage apertures. Experimental results show that the proposed method can obtain relatively high accuracy and greatly reduce the training time of ResNet by using compressed data. When the Compression rate (CR) is 70% and the observation matrix is a Gaussian random matrix, the average accuracy is 96.67% and the training time is only 50% compared to uncompressed data. The research work provides a new intelligent leak identification under different leak apertures using WSN and has important application prospects in saving water resources.

On the Fourier transform of a quantitative trait: Implications for compressive sensing

This paper explores the genotype-phenotype relationship. It outlines conditions under which the dependence of a quantitative trait on the genome might be predictable, based on measurement of a limited subset of genotypes. It uses the theory of real-valued Boolean functions in a systematic way to translate trait data into the Fourier domain. Important trait features, such as the roughness of the trait landscape or the modularity of a trait have a simple Fourier interpretation. Ruggedness at a gene location corresponds to high sensitivity to mutation, while a modular organization of gene activity reduces such sensitivity.Traits where rugged loci are rare will naturally compress gene data in the Fourier domain, leading to a sparse representation of trait data, concentrated in identifiable, low-level coefficients. This Fourier representation of a trait organizes epistasis in a form which is isometric to the trait data. As Fourier matrices are known to be maximally incoherent with the standard basis, this permits employing compressive sensing techniques to work from data sets that are relatively small—sometimes even of polynomial size—compared to the exponentially large sets of possible genomes.This theory provides a theoretical underpinning for systematic use of Boolean function machinery to dissect the dependency of a trait on the genome and environment.

Stratified Stochastic Variational Inference for High-Dimensional Network Factor Model

There has been considerable recent interest in Bayesian modeling of high-dimensional networks via latent space approaches. When the number of nodes increases, estimation based on Markov chain Monte Carlo can be extremely slow and show poor mixing, thereby motivating research on alternative algorithms that scale well in high-dimensional settings. In this article, we focus on the latent factor model, a widely used approach for latent space modeling of network data. We develop scalable algorithms to conduct approximate Bayesian inference via stochastic optimization. Leveraging sparse representations of network data, the proposed algorithms show massive computational and storage benefits, and allow to conduct inference in settings with thousands of nodes. An R package with an efficient c++ implementation of the proposed algorithms is provided.

A Sparse Representation of Social Media, Internet Query, and Surveillance Data to Forecast Dengue Case Number using Hybrid DecompositionBidirectional LSTM

A Sparse Representation of Social Media, Internet Query, and Surveillance Data to Forecast Dengue Case Number using Hybrid DecompositionBidirectional LSTM