Sparse Coding Research Articles

Improving the sparse coding model via hybrid Gaussian priors

Sparse Coding (SC) imposes a sparse prior on the representation coefficients under a dictionary or a sensing matrix. However, the sparse regularization, approximately expressed as the L1-norm, is not strongly convex. The uniqueness of the optimal solution requires the dictionary to be of low mutual coherence. As a specialized form of SC, Convolutional Sparse Coding (CSC) encounters the same issue. Inspired by the Elastic Net, this paper proposes to learn an additional anisotropic Gaussian prior for the sparse codes, thus improving the convexity of the SC problem and enabling the modeling of feature correlation. As a result, the SC problem is modified by the proposed elastic projection. We thereby analyze the effectiveness of the proposed method under the framework of LISTA and demonstrate that this simple technique has the potential to correct bad codes and reduce the error bound, especially in noisy scenarios. Furthermore, we extend this technique to the CSC model for the vision practice of image denoising. Extensive experimental results show that the learned Gaussian prior significantly improves the performance of both the SC and CSC models. Source codes are available at https://github.com/eeejyang/EPCSCNet.

Convolutional sparse coding network for sparse seismic time-frequency representation

Seismic time-frequency (TF) transforms are essential tools in reservoir interpretation and signal processing, particularly for characterizing frequency variations in non-stationary seismic data. Recently, sparse TF transforms, which leverage sparse coding (SC), have gained significant attention in the geosciences due to their ability to achieve high TF resolution. However, the iterative approaches typically employed in sparse TF transforms are computationally intensive, making them impractical for real seismic data analysis. To address this issue, we propose an interpretable convolutional sparse coding (CSC) network to achieve high TF resolution. The proposed model is generated based on the traditional short-time Fourier transform (STFT) transform and a modified UNet, named ULISTANet. In this design, we replace the conventional convolutional layers of the UNet with learnable iterative shrinkage thresholding algorithm (LISTA) blocks, a specialized form of CSC. The LISTA block, which evolves from the traditional iterative shrinkage thresholding algorithm (ISTA), is optimized for extracting sparse features more effectively. Furthermore, we create a synthetic dataset featuring complex frequency-modulated signals to train ULISTANet. Finally, the proposed method's performance is subsequently validated using both synthetic and field data, demonstrating its potential for enhanced seismic data analysis.

Regularization by double complementary priors for full waveform inversion

Full-waveform inversion (FWI) represents an advanced geophysical imaging technique focused on intricately depicting subsurface physical properties by iteratively minimizing the differences between the simulated and observed seismograms. Unfortunately, the conventional FWI utilizing a least-squares loss function suffers from various drawbacks, including the challenge of local minima and the necessity for human intervention in parameter fine-tuning. It is particularly problematic when handling noisy data and inadequate initial models. Recent works have exhibited promising performance in two-dimensional FWI by integrating structural sparse representation to procure adaptive dictionaries. Drawing inspiration from the competitiveness of structural sparse representation, we introduce a paradigm of group sparse residuals that integrates two types of complementary prior information by harnessing both the internal and external subsurface media models. The proposed algorithm is based on an alternate minimization algorithm to guarantee workflow flexibility and efficient optimization capabilities. We experimentally validate our method for two baseline geological models, and a comparison of the results demonstrates that the proposed algorithm faithfully recovers the velocity models and consistently outperforms other traditional or learning-based algorithms. A further benefit from the group sparse coding used in this method is that it reduces the sensitivity to data noise.

A Novel Dictionary Learning Algorithm Based on Prior Knowledge for fMRI Data Analysis

ABSTRACTTask‐based functional magnetic resonance imaging (fMRI) has been widely utilized for brain activation detection and functional network analysis. In recent years, the K‐singular value decomposition (K‐SVD) algorithm has gained increasing attention in the research of fMRI data analysis methods. In this study, we propose a novel temporal feature region‐growing constrained K‐SVD algorithm that incorporates task‐based fMRI temporal prior knowledge and utilizes a region‐growing algorithm to infer potential activation locations. The algorithm incorporates temporal and spatial constraints to enhance the detection of brain activation. Specifically, this paper improves the three stages of the traditional K‐SVD algorithm. First, in the dictionary initialization stage, the automatic target generation process with an independent component analysis algorithm is utilized in conjunction with prior knowledge to enhance the accuracy of initialization. Second, in the sparse coding stage, the region‐growing algorithm is employed to infer potential activation locations based on temporal prior knowledge, thereby imposing spatial constraints to limit the extent of activation regions. Finally, in the dictionary learning stage, soft constraints and low correlation constraints are applied to reinforce the consistency with prior knowledge and enhance the robustness of learning for task‐related atoms. The proposed method was validated on simulated and real fMRI data, showing superior performance in detecting brain activation compared with traditional methods. The results indicate that the algorithm accurately identifies activated brain regions, providing an effective approach for studying brain function in clinical applications.

A Deep Unfolding Network for Multispectral and Hyperspectral Image Fusion

Multispectral and hyperspectral image fusion (MS/HS fusion) aims to generate a high-resolution hyperspectral (HRHS) image by fusing a high-resolution multispectral (HRMS) and a low-resolution hyperspectral (LRHS) images. The deep unfolding-based MS/HS fusion method is a representative deep learning paradigm due to its excellent performance and sufficient interpretability. However, existing deep unfolding-based MS/HS fusion methods only rely on a fixed linear degradation model, which focuses on modeling the relationships between HRHS and HRMS, as well as HRHS and LRHS. In this paper, we break free from this observation model framework and propose a new observation model. Firstly, the proposed observation model is built based on the convolutional sparse coding (CSC) technique, and then a proximal gradient algorithm is designed to solve this model. Secondly, we unfold the iterative algorithm into a deep network, dubbed as MHF-CSCNet, where the proximal operators are learned using convolutional neural networks. Finally, all trainable parameters can be automatically learned end-to-end from the training pairs. Experimental evaluations conducted on various benchmark datasets demonstrate the superiority of our method both quantitatively and qualitatively compared to other state-of-the-art methods.

Encoding human activities using multimodal wearable sensory data

Human Activity Recognition (HAR) is the task to automatically analyze and recognize human body gestures or actions. HAR using time-series multi-modal sensory data is a challenging and important task in the field of machine learning and feature engineering due to its increasing demands in numerous real-world applications such as healthcare, sports and surveillance. Numerous daily wearable devices e.g., smartphones, smartwatches, and smart glasses can be used to collect and analyze the human activities on an unprecedented scale. This paper presents a generic framework to recognize the different human activities using continuous time-series multimodal sensory data of these smart gadgets. The proposed framework follows the channel of Bag-of-Features which consists of four steps: (i) Data acquisition and pre-processing, (ii) codebook computation, (iii) feature encoding, and (iv) classification. Each step in the framework plays a significant role to generate an appropriate feature representation of raw sensory data for efficient activity recognition. In the first step, we employed a simple overlapped-window sampling approach to segment the continuous time-series sensory data to make it suitable for activity recognition. Secondly, we build a codebook using k-means clustering algorithm to group the similar sub-sequences. The center of each group is known as codeword and we assume that it represents a specific movement in the activity sequence. The third step consists of feature encoding which transform the raw sensory data of activity sequence into its respective high-level representation for the classification. Specifically, we presented three reconstruction-based encoding techniques to encode sensory data, namely: Sparse Coding, Local Coordinate Coding, and Locality-constrained Linear Coding. The segmented activity sub-sequences are transformed to high-level representation using these techniques and earlier computed codebook. Finally, the encoded features are classified using a simple Random Forest classifier. The proposed HAR framework using three different encoding techniques is evaluated on three large benchmark datasets: UniMiB-SHAR dataset, MHEALTH dataset and WISDM dataset and their results are compared with most recent state-of-the-art techniques. It outperforms the existing techniques and achieves 98.39%, 99.5%, and 99.4% recognition scores on UniMiB-SHAR dataset, MHEALTH dataset, and WISDM dataset respectively. The excellent recognition results and computational analysis confirms the effectiveness and efficiency of the proposed framework.

A Distributed Non-Intrusive Load Monitoring Method Using Karhunen–Loeve Feature Extraction and an Improved Deep Dictionary

In recent years, the non-invasive load monitoring (NILM) method based on sparse coding has shown promising research prospects. This type of method learns a sparse dictionary for each monitoring target device, and it expresses load decomposition as a problem of signal reconstruction using dictionaries and sparse vectors. The existing NILM methods based on sparse coding have problems such as inability to be applied to multi-state and time-varying devices, single-load characteristics, and poor recognition ability for similar devices in distributed manners. Using the analysis above, this paper focuses on devices with similar features in households and proposes a distributed non-invasive load monitoring method using Karhunen–Loeve (KL) feature extraction and an improved deep dictionary. Firstly, Karhunen–Loeve expansion (KLE) is used to perform subspace expansion on the power waveform of the target device, and a new load feature is extracted by combining singular value decomposition (SVD) dimensionality reduction. Afterwards, the states of all the target devices are modeled as super states, and an improved deep dictionary based on the distance separability measure function (DSM-DDL) is learned for each super state. Among them, the state transition probability matrix and observation probability matrix in the hidden Markov model (HMM) are introduced as the basis for selecting the dictionary order during load decomposition. The KL feature matrix of power observation values and improved depth dictionary are used to discriminate the current super state based on the minimum reconstruction error criterion. The test results based on the UK-DALE dataset show that the KL feature matrix can effectively reduce the load similarity of devices. Combined with DSM-DDL, KL has a certain information acquisition ability and acceptable computational complexity, which can effectively improve the load decomposition accuracy of similar devices, quickly and accurately estimating the working status and power demand of household appliances.

Sparse-Coding Variational Autoencoders.

The sparse coding model posits that the visual system has evolved to efficiently code natural stimuli using a sparse set of features from an overcomplete dictionary. The original sparse coding model suffered from two key limitations, however: (1) computing the neural response to an image patch required minimizing a nonlinear objective function via recurrent dynamics and (2) fitting relied on approximate inference methods that ignored uncertainty. Although subsequent work has developed several methods to overcome these obstacles, we propose a novel solution inspired by the variational autoencoder (VAE) framework. We introduce the sparse coding variational autoencoder (SVAE), which augments the sparse coding model with a probabilistic recognition model parameterized by a deep neural network. This recognition model provides a neurally plausible feedforward implementation for the mapping from image patches to neural activities and enables a principled method for fitting the sparse coding model to data via maximization of the evidence lower bound (ELBO). The SVAE differs from standard VAEs in three key respects: the latent representation is overcomplete (there are more latent dimensions than image pixels), the prior is sparse or heavy-tailed instead of gaussian, and the decoder network is a linear projection instead of a deep network. We fit the SVAE to natural image data under different assumed prior distributions and show that it obtains higher test performance than previous fitting methods. Finally, we examine the response properties of the recognition network and show that it captures important nonlinear properties of neurons in the early visual pathway.

Compilation of Shape Operators on Sparse Arrays

We show how to build a compiler for a sparse array language that supports shape operators such as reshaping or concatenating arrays, in addition to compute operators. Existing sparse array programming systems implement generic shape operators for only some sparse data structures, reduce shape operators on other data structures to those, and do not support fusion. Our system compiles sparse array expressions to code that efficiently iterates over reshaped views of irregular sparse data structures, without needing to materialize temporary storage for intermediates. Our evaluation shows that our approach generates sparse array code competitive with popular sparse array libraries: our generated shape operators achieve geometric mean speed-ups of 1.66×–15.3× when compared to hand-written kernels in scipy.sparse and 1.67×–651× when compared to generic implementations in pydata/sparse. For operators that require data structure conversions in these libraries, our generated code achieves geometric mean speed-ups of 7.29×–13.0× when compared to scipy.sparse and 21.3×–511× when compared to pydata/sparse. Finally, our evaluation demonstrates that fusing shape and compute operators improves the performance of several expressions by geometric mean speed-ups of 1.22×–2.23×.

Dictionary Learning Method Based on K-Sparse Approximation and Orthogonal Procrustes Analysis for Reconstruction in Bioluminescence Tomography.

Bioluminescence tomography (BLT) is one kind of noninvasive optical molecular imaging technology, widely used to study molecular activities and disease progression inside live animals. By combining the optical propagation model and inversion algorithm, BLT enables three-dimensional imaging and quantitative analysis of light sources within organisms. However, challenges like light scattering and absorption in tissues, and the complexity of biological structures, significantly impact the accuracy of BLT reconstructions. Here, we propose a dictionary learning method based on K-sparse approximation and Orthogonal Procrustes analysis (KSAOPA). KSAOPA uses an iterative alternating optimization strategy, enhancing solution sparsity with k-coefficients Lipschitzian mappings for sparsity(K-LIMAPS) in the sparse coding stage, and reducing errors with Orthogonal Procrustes analysis in the dictionary update stage, leading to stable and precise reconstructions. We assessed the method performance through simulations and in vivo experiments, which showed that KSAOPA excels in localization accuracy, morphological recovery, and in vivo applicability compared to other methods.

A model-based deep learning approach to interpretable impact force localization and reconstruction

Model-based and deep learning-based methods have been widely studied for force identification. However, model-based methods usually have high computational complexity and face challenges in parameter setting, while the generic network structures in deep learning methods are mostly designed empirically. In this paper, a model-based deep learning network is proposed to simultaneously localize and reconstruct impact f[orces. The proposed network, dubbed NSC-Net, is obtained by unrolling the iterative shrinkage-thresholding algorithm (ISTA) for the non-negative sparse coding (NSC) model. NSC-Net combines the interpretability of the model-based ISTA with the powerful parameter learning capability of deep learning. The transfer matrix is embedded into the network as physical information through learnable scaling. The structure of NSC-Net is based on explicit theoretical analysis, which enhances its interpretability in terms of structural design. In contrast to ISTA, the parameters in NSC-Net can be trained end-to-end from the training data without manual setting. Simulations and experiments on composite panels are conducted to validate the performance of the proposed method. The results demonstrate that NSC-Net accurately localizes and reconstructs impact forces, outperforming both ISTA and learned iterative shrinkage-thresholding algorithm (LISTA) network. Additionally, NSC-Net exhibits excellent noise immunity. Furthermore, the systematic approach of constructing an algorithm unrolling network for impact force identification is summarized, aiming to facilitate further related research.

Behind-the-Meter Load and PV Disaggregation via Deep Spatiotemporal Graph Generative Sparse Coding With Capsule Network.

Nowadays, rooftop photovoltaic (PV) panels are getting enormous attention as clean and sustainable sources of energy due to the increasing energy demand, depreciating physical assets, and global environmental challenges. In residential areas, the large-scale integration of these generation resources influences the customer load profile and introduces uncertainty to the distribution system's net load. Since such resources are typically located behind the meter (BtM), an accurate estimation of BtM load and PV power will be crucial for distribution network operation. This article proposes the spatiotemporal graph sparse coding (SC) capsule network that incorporates SC into deep generative graph modeling and capsule networks for accurate BtM load and PV generation estimation. A set of neighboring residential units are modeled as a dynamic graph in which the edges represent the correlation among their net demands. A generative encoder-decoder model, i.e., spectral graph convolution (SGC) attention peephole long short-term memory (PLSTM), is devised to extract the highly nonlinear spatiotemporal patterns from the formed dynamic graph. Later, to enrich the latent space sparsity, a dictionary is learned in the hidden layer of the proposed encoder-decoder, and the corresponding sparse codes are procured. Such sparse representation is used by a capsule network to estimate the BtM PV generation and the load of the entire residential units. Experimental results on two real-world energy disaggregation (ED) datasets, Pecan Street and Ausgrid, demonstrate more than 9.8% and 6.3% root mean square error (RMSE) improvements in BtM PV and load estimation over the state-of-the-art, respectively.

Mosaic evolution of a learning and memory circuit in Heliconiini butterflies.

How do neural circuits accommodate changes that produce cognitive variation? We explore this question by analyzing the evolutionary dynamics of an insect learning and memory circuit centered within the mushroom body. Mushroom bodies are composed of a conserved wiring logic, mainly consisting of Kenyon cells, dopaminergic neurons, and mushroom body output neurons. Despite this conserved makeup, there is huge diversity in mushroom body size and shape across insects. However, empirical data on how evolution modifies the function and architecture of this circuit are largely lacking. To address this, we leverage the recent radiation of a Neotropical tribe of butterflies, the Heliconiini (Nymphalidae), which show extensive variation in mushroom body size over comparatively short phylogenetic timescales, linked to specific changes in foraging ecology, life history, and cognition. To understand how such an extensive increase in size is accommodated through changes in lobe circuit architecture, we combined immunostainings of structural markers, neurotransmitters, and neural injections to generate new, quantitative anatomies of the Nymphalid mushroom body lobe. Our comparative analyses across Heliconiini demonstrate that some Kenyon cell sub-populations expanded at higher rates than others in Heliconius and identify an additional increase in GABA-ergic feedback neurons, which are essential for non-elemental learning and sparse coding. Taken together, our results demonstrate mosaic evolution of functionally related neural systems and cell types and identify that evolutionary malleability in an architecturally conserved parallel circuit guides adaptation in cognitive ability.

A deep learning approach for pose error prediction in parallel robots

Parallel robots are increasingly favored in industrial automation due to their high precision, stability, and rigidity in high-speed and heavy-load tasks. However, factors such as manufacturing tolerances, assembly deviations, loads, and environmental conditions can lead to pose inaccuracies, manifesting as both geometric and non-geometric errors. Traditional model-driven approaches effectively predict geometric errors but often neglect non-geometric factors, potentially impacting the overall accuracy. This paper presents a deep learning-based method for comprehensive pose error prediction in parallel robots, addressing both geometric and non-geometric errors. The proposed approach employs an encoder-decoder network that takes target pose inputs and outputs predicted pose errors. The encoder integrates multi-layer convolutional sparse coding (CSC) blocks and efficient channel attention (ECA) modules to extract features from the network inputs. Monte Carlo dropout is utilized to estimate prediction uncertainties, enhancing the reliability of the network predictions. Experimental validation on a Stewart platform demonstrates the high accuracy and robustness of the method under various loading conditions. Comparative analyses with several typical models affirm the superior performance of the proposed network, indicating its potential to improve the operational accuracy and reliability of parallel robots in high-precision applications.

Improved multi-focus image fusion using online convolutional sparse coding based on sample-dependent dictionary

Multi-focus image fusion merges multiple images captured from different focused regions of a scene to create a fully-focused image. Convolutional sparse coding (CSC) methods are commonly employed for accurate extraction of focused regions, but they often disregard computational costs. To overcome this, an online convolutional sparse coding (OCSC) technique was introduced, but its performance is still limited by the number of filters used, affecting overall performance negatively. To address these limitations, a novel approach called Sample-Dependent Dictionary-based Online Convolutional Sparse Coding (SCSC) was proposed. SCSC enables the utilization of additional filters while maintaining low time and space complexity for processing high-dimensional or large data. Leveraging the computational efficiency and effective global feature extraction of SCSC, we propose a novel method for multi-focus image fusion. Our method involves a two-layer decomposition of each source image, yielding a base layer capturing the predominant features and a detail layer containing finer details. The amalgamation of the fused base and detail layers culminates in the reconstruction of the final image. The proposed method significantly mitigates artifacts, preserves fine details at the focus boundary, and demonstrates notable enhancements in both visual quality and objective evaluation of multi-focus image fusion.

Sparse null codes emerge and dominate representations in deep neural network vision models

Sparse null codes emerge and dominate representations in deep neural network vision models

Factorizers for distributed sparse block codes

Distributed sparse block codes (SBCs) exhibit compact representations for encoding and manipulating symbolic data structures using fixed-width vectors. One major challenge however is to disentangle, or factorize, the distributed representation of data structures into their constituent elements without having to search through all possible combinations. This factorization becomes more challenging when SBCs vectors are noisy due to perceptual uncertainty and approximations made by modern neural networks to generate the query SBCs vectors. To address these challenges, we first propose a fast and highly accurate method for factorizing a more flexible and hence generalized form of SBCs, dubbed GSBCs. Our iterative factorizer introduces a threshold-based nonlinear activation, conditional random sampling, and an ℓ ∞ -based similarity metric. Its random sampling mechanism, in combination with the search in superposition, allows us to analytically determine the expected number of decoding iterations, which matches the empirical observations up to the GSBC’s bundling capacity. Secondly, the proposed factorizer maintains a high accuracy when queried by noisy product vectors generated using deep convolutional neural networks (CNNs). This facilitates its application in replacing the large fully connected layer (FCL) in CNNs, whereby C trainable class vectors, or attribute combinations, can be implicitly represented by our factorizer having F-factor codebooks, each with C F fixed codevectors. We provide a methodology to flexibly integrate our factorizer in the classification layer of CNNs with a novel loss function. With this integration, the convolutional layers can generate a noisy product vector that our factorizer can still decode, whereby the decoded factors can have different interpretations based on downstream tasks. We demonstrate the feasibility of our method on four deep CNN architectures over CIFAR-100, ImageNet-1K, and RAVEN datasets. In all use cases, the number of parameters and operations are notably reduced compared to the FCL.

Removing cloud shadows from ground-based solar imagery

The study and prediction of space weather entails the analysis of solar images showing structures of the Sun’s atmosphere. When imaged from the Earth’s ground, images may be polluted by terrestrial clouds which hinder the detection of solar structures. We propose a new method to remove cloud shadows, based on a U-Net architecture, and compare classical supervision with conditional GAN. We evaluate our method on two different imaging modalities, using both real images and a new dataset of synthetic clouds. Quantitative assessments are obtained through image quality indices (RMSE, PSNR, SSIM, and FID). We demonstrate improved results with regards to the traditional cloud removal technique and a sparse coding baseline, on different cloud types and textures.

A novel theoretical analysis on optimal pipeline of multi-frame image super-resolution using sparse coding

Super-resolution is the process of obtaining a high-resolution (HR) image from one or more low-resolution (LR) images. Single image super-resolution (SISR) deals with one LR image while multi-frame super-resolution (MFSR) employs several LR ones to reach the HR output. MFSR pipeline consists of alignment, fusion, and reconstruction. We conduct a theoretical analysis using sparse coding (SC) and iterative shrinkage-thresholding algorithm to fill the gap of mathematical justification in the execution order of the optimal MFSR pipeline. Our analysis recommends executing alignment and fusion before the reconstruction stage (whether through deconvolution or SISR techniques). The suggested order ensures enhanced performance in terms of peak signal-to-noise ratio and structural similarity index. The optimal pipeline also reduces computational complexity compared to intuitive approaches that apply SISR to each input LR image. Also, we demonstrate the usefulness of SC in analysis of computer vision tasks such as MFSR, leveraging the sparsity assumption in natural images. Simulation results support the findings of our theoretical analysis, both quantitatively and qualitatively.

Polar coded SCMA-OCDM based on chaotic scrambled SLM

In this work, the polar coded sparse code multiple access (SCMA) scheme based on chaotic scrambled selective mapping (CS_SLM) is proposed for future passive optical networks (PONs). The original bits are coded by polar and mapped into constellation symbols by sparse code. Then orthogonal chirp division multiplexing (OCDM) modulation is adopted which has better anti-interference ability than orthogonal frequency division multiplexing (OFDM). At the receiver, the joint iterative detection and decoding for polar and SCMA is used to further improve the BER performance. The CS_SLM is helpful to improve the reliability and security, while keeping high spectral efficiency of SCMA-OCDM modulation. A 49.42 Gb/s coded SCMA-OCDM PON over 2 km seven-core fiber is demonstrated in the experiment. The results show that the SCMA-OCDM scheme gets a receiver sensitivity of 0.8 dB at a bit error rate (BER) of 3.8 × 10−3, compared with SCMA-OFDM. Because of peak to average power ratio (PAPR), the polar coded SCMA-OCDM based on CS_SLM is compared with clipping and filtering (CAF) method, about 2.08 dB receiver sensitivity gain is obtained when BER is lower than 10−5. The polar coded SCMA-OCDM based on CS_SLM also takes security into the consideration, due to the downlink broadcasting of PON. It can be a promising candidate for the next generation PONs.